[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 08:21:35.478147 687 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.171.254:34965
I20250626 08:21:35.479552 687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 08:21:35.480540 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:21:35.496672 694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:35.496734 693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:35.497417 696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:36.695370 687 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 08:21:36.695777 695 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 08:21:36.700193 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:21:36.700454 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:21:36.700610 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926096700594 us; error 0 us; skew 500 ppm
I20250626 08:21:36.701418 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:21:36.707856 687 webserver.cc:469] Webserver started at http://127.0.171.254:42903/ using document root <none> and password file <none>
I20250626 08:21:36.708704 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:21:36.708900 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:21:36.709295 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:21:36.715165 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/master-0-root/instance:
uuid: "8c26721dcad5495fb4c11659eb02d91d"
format_stamp: "Formatted at 2025-06-26 08:21:36 on dist-test-slave-660p"
I20250626 08:21:36.723017 687 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250626 08:21:36.728474 702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:36.729544 687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250626 08:21:36.729861 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/master-0-root
uuid: "8c26721dcad5495fb4c11659eb02d91d"
format_stamp: "Formatted at 2025-06-26 08:21:36 on dist-test-slave-660p"
I20250626 08:21:36.730217 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:21:36.791739 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:21:36.793215 687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 08:21:36.793625 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:21:36.865207 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.254:34965
I20250626 08:21:36.865445 753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.254:34965 every 8 connection(s)
I20250626 08:21:36.872268 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:36.893635 754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: Bootstrap starting.
I20250626 08:21:36.899274 754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:36.901379 754 log.cc:826] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: Log is configured to *not* fsync() on all Append() calls
I20250626 08:21:36.905728 754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: No bootstrap required, opened a new log
I20250626 08:21:36.926462 754 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER }
I20250626 08:21:36.927014 754 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:36.927295 754 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c26721dcad5495fb4c11659eb02d91d, State: Initialized, Role: FOLLOWER
I20250626 08:21:36.928071 754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER }
I20250626 08:21:36.928582 754 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:21:36.928843 754 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:21:36.929167 754 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:36.933270 754 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER }
I20250626 08:21:36.934010 754 leader_election.cc:304] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c26721dcad5495fb4c11659eb02d91d; no voters:
I20250626 08:21:36.935837 754 leader_election.cc:290] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:21:36.936161 757 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:36.938336 757 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 1 LEADER]: Becoming Leader. State: Replica: 8c26721dcad5495fb4c11659eb02d91d, State: Running, Role: LEADER
I20250626 08:21:36.939282 757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER }
I20250626 08:21:36.940415 754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:21:36.950253 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c26721dcad5495fb4c11659eb02d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER } }
I20250626 08:21:36.951073 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [sys.catalog]: This master's current role is: LEADER
I20250626 08:21:36.950773 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c26721dcad5495fb4c11659eb02d91d. Latest consensus state: current_term: 1 leader_uuid: "8c26721dcad5495fb4c11659eb02d91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c26721dcad5495fb4c11659eb02d91d" member_type: VOTER } }
I20250626 08:21:36.951593 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [sys.catalog]: This master's current role is: LEADER
I20250626 08:21:36.954142 766 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:21:36.967213 766 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:21:36.969039 687 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 08:21:36.983964 766 catalog_manager.cc:1349] Generated new cluster ID: 4b8b7dace288454f9af0c6f05f565db1
I20250626 08:21:36.984288 766 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:21:37.010169 766 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:21:37.012295 766 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:21:37.032322 766 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: Generated new TSK 0
I20250626 08:21:37.033263 766 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:21:37.100713 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:21:37.109977 776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:37.111734 777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:37.114764 779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:37.115510 687 server_base.cc:1048] running on GCE node
I20250626 08:21:37.116250 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:21:37.116456 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:21:37.116621 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926097116602 us; error 0 us; skew 500 ppm
I20250626 08:21:37.117153 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:21:37.121592 687 webserver.cc:469] Webserver started at http://127.0.171.193:41173/ using document root <none> and password file <none>
I20250626 08:21:37.122135 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:21:37.122335 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:21:37.122598 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:21:37.123685 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-0-root/instance:
uuid: "e94e795053dd4dd7aeb473e394a14400"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.129314 687 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 08:21:37.133504 784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.134341 687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250626 08:21:37.134686 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-0-root
uuid: "e94e795053dd4dd7aeb473e394a14400"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.135063 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:21:37.156340 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:21:37.158324 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:21:37.161108 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:21:37.166646 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:21:37.166968 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.167312 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:21:37.167543 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.256829 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.193:36347
I20250626 08:21:37.256877 846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.193:36347 every 8 connection(s)
I20250626 08:21:37.264670 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:21:37.275797 851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:37.283563 852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:37.295341 847 heartbeater.cc:344] Connected to a master server at 127.0.171.254:34965
I20250626 08:21:37.295859 847 heartbeater.cc:461] Registering TS with master...
I20250626 08:21:37.297060 847 heartbeater.cc:507] Master 127.0.171.254:34965 requested a full tablet report, sending...
W20250626 08:21:37.297127 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:37.297698 687 server_base.cc:1048] running on GCE node
I20250626 08:21:37.298476 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:21:37.298755 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:21:37.299000 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926097298977 us; error 0 us; skew 500 ppm
I20250626 08:21:37.299769 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:21:37.300581 719 ts_manager.cc:194] Registered new tserver with Master: e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347)
I20250626 08:21:37.304401 719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40122
I20250626 08:21:37.308868 687 webserver.cc:469] Webserver started at http://127.0.171.194:40525/ using document root <none> and password file <none>
I20250626 08:21:37.309382 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:21:37.309576 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:21:37.309839 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:21:37.310954 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-1-root/instance:
uuid: "5e244cde94294d2887a4fa403b6b25db"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.316417 687 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250626 08:21:37.320731 859 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.321549 687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250626 08:21:37.321897 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-1-root
uuid: "5e244cde94294d2887a4fa403b6b25db"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.322263 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:21:37.349812 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:21:37.351488 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:21:37.353986 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:21:37.357674 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:21:37.357951 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250626 08:21:37.358237 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:21:37.358448 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.444087 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.194:43447
I20250626 08:21:37.444172 921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.194:43447 every 8 connection(s)
I20250626 08:21:37.451339 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:21:37.458784 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:21:37.461035 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:37.466256 922 heartbeater.cc:344] Connected to a master server at 127.0.171.254:34965
I20250626 08:21:37.466677 922 heartbeater.cc:461] Registering TS with master...
I20250626 08:21:37.467643 922 heartbeater.cc:507] Master 127.0.171.254:34965 requested a full tablet report, sending...
I20250626 08:21:37.469836 719 ts_manager.cc:194] Registered new tserver with Master: 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:37.471611 719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40124
I20250626 08:21:37.478220 687 server_base.cc:1048] running on GCE node
W20250626 08:21:37.479902 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:21:37.480847 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:21:37.481041 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:21:37.481210 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926097481191 us; error 0 us; skew 500 ppm
I20250626 08:21:37.481725 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:21:37.484690 687 webserver.cc:469] Webserver started at http://127.0.171.195:35587/ using document root <none> and password file <none>
I20250626 08:21:37.485138 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:21:37.485314 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:21:37.485570 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:21:37.486631 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-2-root/instance:
uuid: "bd83cd09aff945b78947107f98f030ef"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.491036 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250626 08:21:37.494588 933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.495436 687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250626 08:21:37.495755 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-2-root
uuid: "bd83cd09aff945b78947107f98f030ef"
format_stamp: "Formatted at 2025-06-26 08:21:37 on dist-test-slave-660p"
I20250626 08:21:37.496094 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1750926095312382-687-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:21:37.532164 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:21:37.533818 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:21:37.535467 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:21:37.538508 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:21:37.538776 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.539028 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:21:37.539209 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:21:37.623741 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.195:35605
I20250626 08:21:37.623813 995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.195:35605 every 8 connection(s)
I20250626 08:21:37.640419 996 heartbeater.cc:344] Connected to a master server at 127.0.171.254:34965
I20250626 08:21:37.640785 996 heartbeater.cc:461] Registering TS with master...
I20250626 08:21:37.641634 996 heartbeater.cc:507] Master 127.0.171.254:34965 requested a full tablet report, sending...
I20250626 08:21:37.643498 719 ts_manager.cc:194] Registered new tserver with Master: bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:37.643803 687 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014646009s
I20250626 08:21:37.645227 719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40130
I20250626 08:21:37.647018 687 test_util.cc:276] Using random seed: 1075053965
I20250626 08:21:37.702617 719 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40132:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 08:21:37.707540 719 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:21:37.826736 887 tablet_service.cc:1468] Processing CreateTablet for tablet da500eca704a4606932806ffc16cc0d6 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250626 08:21:37.828367 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da500eca704a4606932806ffc16cc0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.828410 961 tablet_service.cc:1468] Processing CreateTablet for tablet da500eca704a4606932806ffc16cc0d6 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250626 08:21:37.829736 961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da500eca704a4606932806ffc16cc0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.830963 960 tablet_service.cc:1468] Processing CreateTablet for tablet 3501d034516444b687806dd35b56f68f (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250626 08:21:37.832260 960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3501d034516444b687806dd35b56f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.835139 958 tablet_service.cc:1468] Processing CreateTablet for tablet 1b4d127c6ba04cf185518373d93861d4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250626 08:21:37.836455 958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b4d127c6ba04cf185518373d93861d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.838404 957 tablet_service.cc:1468] Processing CreateTablet for tablet 89e7562cf02a4197b81a93ff05bb5934 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250626 08:21:37.839706 957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e7562cf02a4197b81a93ff05bb5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.845147 955 tablet_service.cc:1468] Processing CreateTablet for tablet 682f301287ec4cffbc0d74c652054cb4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250626 08:21:37.846899 955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682f301287ec4cffbc0d74c652054cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.840626 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3501d034516444b687806dd35b56f68f (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250626 08:21:37.861359 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3501d034516444b687806dd35b56f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.856482 953 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2c8fbc740e4c80bf8252ae259d5280 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250626 08:21:37.856088 954 tablet_service.cc:1468] Processing CreateTablet for tablet d6f17446740b4a6e83973b82c26d0b7a (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250626 08:21:37.850883 959 tablet_service.cc:1468] Processing CreateTablet for tablet 36cf77b4b10d48468e830953d3c5b450 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250626 08:21:37.869673 953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2c8fbc740e4c80bf8252ae259d5280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.863369 956 tablet_service.cc:1468] Processing CreateTablet for tablet 0df58552e5234a61a6385b169f030d3d (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250626 08:21:37.862643 883 tablet_service.cc:1468] Processing CreateTablet for tablet 89e7562cf02a4197b81a93ff05bb5934 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250626 08:21:37.859931 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1b4d127c6ba04cf185518373d93861d4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250626 08:21:37.880486 956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df58552e5234a61a6385b169f030d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.884140 880 tablet_service.cc:1468] Processing CreateTablet for tablet d6f17446740b4a6e83973b82c26d0b7a (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250626 08:21:37.888264 812 tablet_service.cc:1468] Processing CreateTablet for tablet da500eca704a4606932806ffc16cc0d6 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250626 08:21:37.890651 812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da500eca704a4606932806ffc16cc0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.872627 881 tablet_service.cc:1468] Processing CreateTablet for tablet 682f301287ec4cffbc0d74c652054cb4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250626 08:21:37.895306 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0df58552e5234a61a6385b169f030d3d (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250626 08:21:37.898227 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682f301287ec4cffbc0d74c652054cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.874264 885 tablet_service.cc:1468] Processing CreateTablet for tablet 36cf77b4b10d48468e830953d3c5b450 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250626 08:21:37.899214 959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cf77b4b10d48468e830953d3c5b450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.894064 879 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2c8fbc740e4c80bf8252ae259d5280 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250626 08:21:37.908859 811 tablet_service.cc:1468] Processing CreateTablet for tablet 3501d034516444b687806dd35b56f68f (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250626 08:21:37.911079 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b4d127c6ba04cf185518373d93861d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.917299 811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3501d034516444b687806dd35b56f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.918767 810 tablet_service.cc:1468] Processing CreateTablet for tablet 36cf77b4b10d48468e830953d3c5b450 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250626 08:21:37.921530 810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cf77b4b10d48468e830953d3c5b450. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.925513 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df58552e5234a61a6385b169f030d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.934405 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2c8fbc740e4c80bf8252ae259d5280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.919422 809 tablet_service.cc:1468] Processing CreateTablet for tablet 1b4d127c6ba04cf185518373d93861d4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250626 08:21:37.939046 806 tablet_service.cc:1468] Processing CreateTablet for tablet 682f301287ec4cffbc0d74c652054cb4 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250626 08:21:37.945708 808 tablet_service.cc:1468] Processing CreateTablet for tablet 89e7562cf02a4197b81a93ff05bb5934 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250626 08:21:37.948318 807 tablet_service.cc:1468] Processing CreateTablet for tablet 0df58552e5234a61a6385b169f030d3d (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250626 08:21:37.945325 954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f17446740b4a6e83973b82c26d0b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.942217 805 tablet_service.cc:1468] Processing CreateTablet for tablet d6f17446740b4a6e83973b82c26d0b7a (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250626 08:21:37.954124 809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b4d127c6ba04cf185518373d93861d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.949927 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f17446740b4a6e83973b82c26d0b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:37.942093 804 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2c8fbc740e4c80bf8252ae259d5280 (DEFAULT_TABLE table=test-workload [id=b01f9bf02ee741e1adf3f60da22199ec]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250626 08:21:37.979888 1014 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:37.999040 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36cf77b4b10d48468e830953d3c5b450. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:21:37.999225 767 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:21:38.006362 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e7562cf02a4197b81a93ff05bb5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.015653 808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89e7562cf02a4197b81a93ff05bb5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.035789 805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f17446740b4a6e83973b82c26d0b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.065212 807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df58552e5234a61a6385b169f030d3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.074083 806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 682f301287ec4cffbc0d74c652054cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.075182 1015 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.090113 804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2c8fbc740e4c80bf8252ae259d5280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:21:38.113965 1015 tablet_bootstrap.cc:654] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.137146 1016 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.143319 1014 tablet_bootstrap.cc:654] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.148201 1015 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.148726 1015 ts_tablet_manager.cc:1397] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.006s
I20250626 08:21:38.153998 1015 raft_consensus.cc:357] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.155318 1015 raft_consensus.cc:383] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.155877 1015 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.156919 1015 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.172969 1014 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.173815 922 heartbeater.cc:499] Master 127.0.171.254:34965 was elected leader, sending a full tablet report...
I20250626 08:21:38.177618 1014 ts_tablet_manager.cc:1397] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.243s user 0.012s sys 0.013s
I20250626 08:21:38.182639 1015 ts_tablet_manager.cc:1428] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.034s user 0.011s sys 0.006s
I20250626 08:21:38.183861 1015 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.183205 1014 raft_consensus.cc:357] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.184533 1014 raft_consensus.cc:383] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.184887 1014 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.185789 1014 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.189764 1016 tablet_bootstrap.cc:654] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.190514 1015 tablet_bootstrap.cc:654] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
W20250626 08:21:38.202378 923 tablet.cc:2378] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:21:38.203596 996 heartbeater.cc:499] Master 127.0.171.254:34965 was elected leader, sending a full tablet report...
I20250626 08:21:38.205655 1014 ts_tablet_manager.cc:1428] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.028s user 0.009s sys 0.005s
I20250626 08:21:38.206648 1014 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.213364 1014 tablet_bootstrap.cc:654] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.215555 1016 tablet_bootstrap.cc:492] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.216019 1016 ts_tablet_manager.cc:1397] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.121s user 0.015s sys 0.000s
I20250626 08:21:38.218904 1016 raft_consensus.cc:357] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.219640 1016 raft_consensus.cc:383] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.219933 1016 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.220786 1016 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.227627 1015 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.228266 1015 ts_tablet_manager.cc:1397] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.005s
I20250626 08:21:38.231834 1015 raft_consensus.cc:357] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.232806 1015 raft_consensus.cc:383] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.233184 1015 raft_consensus.cc:738] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.236737 1014 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.237188 1014 ts_tablet_manager.cc:1397] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.004s
I20250626 08:21:38.239828 1014 raft_consensus.cc:357] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.240520 1014 raft_consensus.cc:383] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.240818 1014 raft_consensus.cc:738] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.241678 1014 consensus_queue.cc:260] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.243870 1014 ts_tablet_manager.cc:1428] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 08:21:38.244717 1014 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.250943 1014 tablet_bootstrap.cc:654] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.259832 1015 consensus_queue.cc:260] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.262436 847 heartbeater.cc:499] Master 127.0.171.254:34965 was elected leader, sending a full tablet report...
I20250626 08:21:38.262827 1014 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.263099 1016 ts_tablet_manager.cc:1428] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.047s user 0.023s sys 0.022s
I20250626 08:21:38.263284 1014 ts_tablet_manager.cc:1397] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20250626 08:21:38.264012 1016 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.266458 1014 raft_consensus.cc:357] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.267167 1014 raft_consensus.cc:383] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.267495 1014 raft_consensus.cc:738] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.268316 1014 consensus_queue.cc:260] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.270623 1014 ts_tablet_manager.cc:1428] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:21:38.271559 1014 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.271991 1016 tablet_bootstrap.cc:654] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.277973 1014 tablet_bootstrap.cc:654] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.282660 1015 ts_tablet_manager.cc:1428] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.053s user 0.007s sys 0.002s
I20250626 08:21:38.289067 1014 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.289552 1014 ts_tablet_manager.cc:1397] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20250626 08:21:38.292162 1014 raft_consensus.cc:357] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.292862 1014 raft_consensus.cc:383] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.293061 1015 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.293150 1014 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.294410 1014 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.297027 1014 ts_tablet_manager.cc:1428] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 08:21:38.297966 1014 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.299721 1016 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.300204 1016 ts_tablet_manager.cc:1397] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.004s
I20250626 08:21:38.302103 1015 tablet_bootstrap.cc:654] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.303027 1016 raft_consensus.cc:357] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.303790 1016 raft_consensus.cc:383] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.304112 1016 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.304399 1014 tablet_bootstrap.cc:654] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.304839 1016 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.306639 1019 raft_consensus.cc:491] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.307437 1016 ts_tablet_manager.cc:1428] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250626 08:21:38.307126 1019 raft_consensus.cc:513] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.308377 1016 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.317689 1014 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.318318 1014 ts_tablet_manager.cc:1397] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20250626 08:21:38.319535 1019 leader_election.cc:290] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.330725 1020 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.331600 1020 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.335460 1016 tablet_bootstrap.cc:654] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.337934 1020 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.340405 1015 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.340114 1014 raft_consensus.cc:357] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.340947 1015 ts_tablet_manager.cc:1397] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.016s
I20250626 08:21:38.341087 1014 raft_consensus.cc:383] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.341571 1014 raft_consensus.cc:738] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.342660 1014 consensus_queue.cc:260] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.346288 1015 raft_consensus.cc:357] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.351909 1015 raft_consensus.cc:383] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.353407 1015 raft_consensus.cc:738] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.355312 1015 consensus_queue.cc:260] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.363149 1014 ts_tablet_manager.cc:1428] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.044s user 0.008s sys 0.001s
I20250626 08:21:38.366205 1015 ts_tablet_manager.cc:1428] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.025s user 0.001s sys 0.007s
I20250626 08:21:38.367264 1015 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.370348 1020 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.373745 1020 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.375290 1016 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.375674 1020 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.376184 1016 ts_tablet_manager.cc:1397] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.068s user 0.013s sys 0.013s
I20250626 08:21:38.379668 1014 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.386088 1015 tablet_bootstrap.cc:654] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.391850 1016 raft_consensus.cc:357] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.393729 1016 raft_consensus.cc:383] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.394079 1016 raft_consensus.cc:738] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.394369 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.395596 822 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:38.395666 1016 consensus_queue.cc:260] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.403859 1014 tablet_bootstrap.cc:654] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.399845 861 leader_election.cc:304] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.405287 1019 raft_consensus.cc:2802] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.407758 1016 ts_tablet_manager.cc:1428] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250626 08:21:38.408737 1016 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.405751 1019 raft_consensus.cc:491] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.413398 1019 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.429548 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.430866 822 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 0.
I20250626 08:21:38.431146 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.433949 1016 tablet_bootstrap.cc:654] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.434123 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.435127 971 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:38.435374 1019 raft_consensus.cc:513] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.439644 935 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.440713 1020 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.440914 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
W20250626 08:21:38.442142 935 leader_election.cc:343] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.442484 821 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.441763 1020 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.443737 1020 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.447276 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.450587 971 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.446699 1019 leader_election.cc:290] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.461602 971 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:38.463394 863 leader_election.cc:304] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters:
I20250626 08:21:38.465895 1019 raft_consensus.cc:2802] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.461674 821 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:38.467290 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.469264 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.468251 1015 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.472154 1015 ts_tablet_manager.cc:1397] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.105s user 0.014s sys 0.009s
W20250626 08:21:38.471881 935 leader_election.cc:343] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.477648 1014 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.478142 1014 ts_tablet_manager.cc:1397] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.007s
W20250626 08:21:38.478057 935 leader_election.cc:343] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.478696 935 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd83cd09aff945b78947107f98f030ef; no voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.478752 1015 raft_consensus.cc:357] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.475800 1020 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.480916 1015 raft_consensus.cc:383] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.481513 1015 raft_consensus.cc:738] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.484342 1015 consensus_queue.cc:260] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.485441 1014 raft_consensus.cc:357] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.486289 1014 raft_consensus.cc:383] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.486763 1015 ts_tablet_manager.cc:1428] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250626 08:21:38.486811 1014 raft_consensus.cc:738] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.486328 1020 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.486248 1016 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.488159 1040 raft_consensus.cc:491] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.488772 1016 ts_tablet_manager.cc:1397] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.020s
I20250626 08:21:38.488884 1040 raft_consensus.cc:513] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.489632 1020 raft_consensus.cc:2747] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.493114 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
W20250626 08:21:38.494765 861 leader_election.cc:343] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.498037 1019 raft_consensus.cc:695] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:38.501690 1015 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.499212 1019 consensus_queue.cc:237] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.501791 1014 consensus_queue.cc:260] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.505894 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:38.506610 821 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.506986 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:38.505954 1016 raft_consensus.cc:357] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.509092 1016 raft_consensus.cc:383] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.509444 1016 raft_consensus.cc:738] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.511289 1024 raft_consensus.cc:491] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.512487 1024 raft_consensus.cc:513] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.511672 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.518302 821 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 1.
I20250626 08:21:38.518678 971 raft_consensus.cc:2466] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:38.520191 1024 leader_election.cc:290] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.520143 719 catalog_manager.cc:5582] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 1 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.510788 1016 consensus_queue.cc:260] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.525946 1015 tablet_bootstrap.cc:654] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.499325 1040 leader_election.cc:290] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.528183 1014 ts_tablet_manager.cc:1428] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.050s user 0.009s sys 0.000s
W20250626 08:21:38.511960 935 leader_election.cc:343] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.535905 1016 ts_tablet_manager.cc:1428] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.047s user 0.011s sys 0.001s
I20250626 08:21:38.536978 1016 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.537112 863 leader_election.cc:304] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters: e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.530400 1014 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.541879 1040 raft_consensus.cc:2802] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.542375 1040 raft_consensus.cc:491] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.542734 1040 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.545107 1019 raft_consensus.cc:491] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.545585 1019 raft_consensus.cc:513] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.547763 1019 leader_election.cc:290] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.550037 935 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters: 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:38.554420 1042 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.556032 1042 raft_consensus.cc:695] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:38.559119 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.558799 1042 consensus_queue.cc:237] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
W20250626 08:21:38.565698 863 leader_election.cc:343] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.567076 1014 tablet_bootstrap.cc:654] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.570709 1040 raft_consensus.cc:513] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.576645 1040 leader_election.cc:290] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.577550 1016 tablet_bootstrap.cc:654] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.577400 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.578387 971 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.583364 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.588491 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
W20250626 08:21:38.589787 861 leader_election.cc:343] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.590340 861 leader_election.cc:304] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db; no voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400
W20250626 08:21:38.591696 861 leader_election.cc:343] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.595480 1015 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.595983 1015 ts_tablet_manager.cc:1397] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.003s
I20250626 08:21:38.596621 1040 raft_consensus.cc:2747] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.599010 971 raft_consensus.cc:2466] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:38.600075 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.600555 863 leader_election.cc:304] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters: e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.600729 896 raft_consensus.cc:2466] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:38.601306 1015 raft_consensus.cc:357] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.602062 1015 raft_consensus.cc:383] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.602391 1015 raft_consensus.cc:738] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.602483 786 leader_election.cc:304] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.603106 1024 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.603374 1040 raft_consensus.cc:2802] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.603873 1040 raft_consensus.cc:695] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:38.603668 1024 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.603967 1015 consensus_queue.cc:260] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.604732 1040 consensus_queue.cc:237] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.605655 1024 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.606788 1024 raft_consensus.cc:2802] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.607122 1024 raft_consensus.cc:491] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.607494 1024 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.608081 718 catalog_manager.cc:5582] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 0 to 1, leader changed from <none> to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 1 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:38.612433 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.612783 1015 ts_tablet_manager.cc:1428] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20250626 08:21:38.613163 970 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:38.613756 1015 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.614661 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.614617 788 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.617728 1019 raft_consensus.cc:491] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.618273 1053 raft_consensus.cc:2802] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.618854 1019 raft_consensus.cc:513] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.619469 1053 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.620473 1053 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.622135 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
W20250626 08:21:38.623023 788 leader_election.cc:343] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.620865 1019 leader_election.cc:290] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
W20250626 08:21:38.623591 863 leader_election.cc:343] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.625617 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.625967 1015 tablet_bootstrap.cc:654] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
W20250626 08:21:38.627534 786 leader_election.cc:343] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.627631 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
W20250626 08:21:38.629012 861 leader_election.cc:343] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.628887 1024 raft_consensus.cc:513] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.629709 861 leader_election.cc:304] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db; no voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.630885 1019 raft_consensus.cc:2747] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.632632 1042 raft_consensus.cc:491] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.633157 1042 raft_consensus.cc:513] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.635308 1042 leader_election.cc:290] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.641446 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.646303 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.646723 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.644165 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:38.649045 897 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.643440 1024 leader_election.cc:290] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
W20250626 08:21:38.648355 935 leader_election.cc:343] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:21:38.648011 788 leader_election.cc:343] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:21:38.650933 935 leader_election.cc:343] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.652177 935 leader_election.cc:304] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd83cd09aff945b78947107f98f030ef; no voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.654067 1042 raft_consensus.cc:2747] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.654784 1015 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.656700 719 catalog_manager.cc:5582] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 1 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.658159 1015 ts_tablet_manager.cc:1397] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.007s
I20250626 08:21:38.660094 897 raft_consensus.cc:2466] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:38.661641 1016 tablet_bootstrap.cc:492] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.661391 786 leader_election.cc:304] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters: bd83cd09aff945b78947107f98f030ef
I20250626 08:21:38.662143 1016 ts_tablet_manager.cc:1397] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.008s
I20250626 08:21:38.662331 1024 raft_consensus.cc:2802] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.659523 1014 tablet_bootstrap.cc:492] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.664841 1014 ts_tablet_manager.cc:1397] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.135s user 0.029s sys 0.011s
I20250626 08:21:38.663771 1053 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.666823 1015 raft_consensus.cc:357] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.668182 1016 raft_consensus.cc:357] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.668906 1016 raft_consensus.cc:383] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.667594 1015 raft_consensus.cc:383] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.669205 1016 raft_consensus.cc:738] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.669688 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:38.669443 1015 raft_consensus.cc:738] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.669970 1016 consensus_queue.cc:260] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
W20250626 08:21:38.671090 786 leader_election.cc:343] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.670763 1015 consensus_queue.cc:260] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.674124 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.674685 971 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.675683 1015 ts_tablet_manager.cc:1428] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250626 08:21:38.676648 1015 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.681500 1042 raft_consensus.cc:491] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.681509 1053 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.682166 1042 raft_consensus.cc:513] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.681582 1014 raft_consensus.cc:357] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.683991 1014 raft_consensus.cc:383] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.684970 1014 raft_consensus.cc:738] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.685736 1014 consensus_queue.cc:260] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.684094 1042 leader_election.cc:290] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.684566 971 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:38.686163 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.687469 788 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters: 5e244cde94294d2887a4fa403b6b25db
W20250626 08:21:38.687671 935 leader_election.cc:343] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.688655 1053 raft_consensus.cc:2802] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.689138 1053 raft_consensus.cc:695] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:38.690009 1053 consensus_queue.cc:237] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.693496 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.690546 1015 tablet_bootstrap.cc:654] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.691234 1024 raft_consensus.cc:695] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:38.695072 1024 consensus_queue.cc:237] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.693194 1014 ts_tablet_manager.cc:1428] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20250626 08:21:38.697328 1014 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
W20250626 08:21:38.697824 935 leader_election.cc:343] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:21:38.698330 935 leader_election.cc:304] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd83cd09aff945b78947107f98f030ef; no voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.699323 1042 raft_consensus.cc:2747] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.705040 1014 tablet_bootstrap.cc:654] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.717571 1015 tablet_bootstrap.cc:492] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.718115 1015 ts_tablet_manager.cc:1397] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.001s
I20250626 08:21:38.721292 1015 raft_consensus.cc:357] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.722074 1015 raft_consensus.cc:383] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.722364 1015 raft_consensus.cc:738] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.723076 1015 consensus_queue.cc:260] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.722471 719 catalog_manager.cc:5582] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.732594 1015 ts_tablet_manager.cc:1428] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250626 08:21:38.737007 1014 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.737632 1014 ts_tablet_manager.cc:1397] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.007s
I20250626 08:21:38.739852 1015 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.740334 1014 raft_consensus.cc:357] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.741091 1014 raft_consensus.cc:383] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.741408 1014 raft_consensus.cc:738] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.742221 1014 consensus_queue.cc:260] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.742491 719 catalog_manager.cc:5582] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.746346 1016 ts_tablet_manager.cc:1428] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.084s user 0.013s sys 0.018s
I20250626 08:21:38.747042 1015 tablet_bootstrap.cc:654] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.747376 1016 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.751258 1014 ts_tablet_manager.cc:1428] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20250626 08:21:38.759096 1014 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: Bootstrap starting.
I20250626 08:21:38.761023 1016 tablet_bootstrap.cc:654] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.764123 1015 tablet_bootstrap.cc:492] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.764541 1015 ts_tablet_manager.cc:1397] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250626 08:21:38.765077 1014 tablet_bootstrap.cc:654] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.766995 1015 raft_consensus.cc:357] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.767707 1015 raft_consensus.cc:383] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.767979 1015 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.768548 1015 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.772106 1016 tablet_bootstrap.cc:492] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.772567 1016 ts_tablet_manager.cc:1397] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.004s
I20250626 08:21:38.776017 1016 raft_consensus.cc:357] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.776953 1016 raft_consensus.cc:383] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.777334 1016 raft_consensus.cc:738] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.778851 1015 ts_tablet_manager.cc:1428] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250626 08:21:38.778239 1016 consensus_queue.cc:260] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.781239 1016 ts_tablet_manager.cc:1428] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20250626 08:21:38.784535 1015 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: Bootstrap starting.
I20250626 08:21:38.787631 1016 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.791095 1015 tablet_bootstrap.cc:654] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.794926 1016 tablet_bootstrap.cc:654] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.795380 1042 raft_consensus.cc:491] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.795892 1042 raft_consensus.cc:513] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.798064 1042 leader_election.cc:290] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:38.799714 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.800057 1014 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: No bootstrap required, opened a new log
I20250626 08:21:38.800513 1014 ts_tablet_manager.cc:1397] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.008s
I20250626 08:21:38.801054 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.801839 897 raft_consensus.cc:2391] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd83cd09aff945b78947107f98f030ef in current term 1: Already voted for candidate e94e795053dd4dd7aeb473e394a14400 in this term.
I20250626 08:21:38.802624 1015 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: No bootstrap required, opened a new log
I20250626 08:21:38.803531 1015 ts_tablet_manager.cc:1397] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20250626 08:21:38.803689 935 leader_election.cc:304] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd83cd09aff945b78947107f98f030ef; no voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.803526 1014 raft_consensus.cc:357] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.804370 1014 raft_consensus.cc:383] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.804629 1014 raft_consensus.cc:738] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.804944 1042 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.805560 1014 consensus_queue.cc:260] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.806334 1015 raft_consensus.cc:357] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.807150 1015 raft_consensus.cc:383] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.807516 1015 raft_consensus.cc:738] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.809336 1014 ts_tablet_manager.cc:1428] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250626 08:21:38.811003 1015 consensus_queue.cc:260] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.812497 1016 tablet_bootstrap.cc:492] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.812942 1016 ts_tablet_manager.cc:1397] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.003s
I20250626 08:21:38.813104 1015 ts_tablet_manager.cc:1428] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250626 08:21:38.815900 1040 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.815754 1016 raft_consensus.cc:357] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.816462 1016 raft_consensus.cc:383] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.816365 1040 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.816820 1016 raft_consensus.cc:738] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.817706 1016 consensus_queue.cc:260] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.818833 1040 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.820303 1016 ts_tablet_manager.cc:1428] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250626 08:21:38.820960 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.821267 1016 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.821699 821 raft_consensus.cc:2391] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e244cde94294d2887a4fa403b6b25db in current term 1: Already voted for candidate bd83cd09aff945b78947107f98f030ef in this term.
I20250626 08:21:38.823760 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.825623 863 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db; no voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.827721 1016 tablet_bootstrap.cc:654] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.828116 1040 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.837154 1042 raft_consensus.cc:2747] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.840322 1058 raft_consensus.cc:491] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.840843 1058 raft_consensus.cc:513] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.843021 1058 leader_election.cc:290] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.844414 1040 raft_consensus.cc:2747] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:38.844267 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:38.845183 897 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:38.846207 786 leader_election.cc:304] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.846335 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.846909 971 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:38.846989 1058 raft_consensus.cc:2802] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.847402 1058 raft_consensus.cc:491] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.847734 1058 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.853605 1016 tablet_bootstrap.cc:492] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.854020 1016 ts_tablet_manager.cc:1397] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.020s
I20250626 08:21:38.854267 1058 raft_consensus.cc:513] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.856199 1058 leader_election.cc:290] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.856487 1016 raft_consensus.cc:357] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.857120 1016 raft_consensus.cc:383] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.857573 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.857690 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:38.858022 1016 raft_consensus.cc:738] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.859241 971 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.862484 1016 consensus_queue.cc:260] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.859416 897 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.864552 1016 ts_tablet_manager.cc:1428] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250626 08:21:38.865610 1040 raft_consensus.cc:491] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.865738 1016 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: Bootstrap starting.
I20250626 08:21:38.866068 1040 raft_consensus.cc:513] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.867882 1040 leader_election.cc:290] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.871215 897 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:38.872689 1016 tablet_bootstrap.cc:654] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: Neither blocks nor log segments found. Creating new log.
I20250626 08:21:38.873054 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.873713 970 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:38.874986 863 leader_election.cc:304] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters:
I20250626 08:21:38.876075 1040 raft_consensus.cc:2802] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:38.876552 1040 raft_consensus.cc:491] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:38.876657 786 leader_election.cc:304] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.877039 1040 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.877473 1024 raft_consensus.cc:2802] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.877962 1024 raft_consensus.cc:695] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:38.878805 1024 consensus_queue.cc:237] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.881626 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.882438 821 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:38.886436 971 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:38.892375 718 catalog_manager.cc:5582] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.894595 1040 raft_consensus.cc:513] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.900339 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:38.901801 821 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.903439 1040 leader_election.cc:290] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.907739 1016 tablet_bootstrap.cc:492] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: No bootstrap required, opened a new log
I20250626 08:21:38.908203 1016 ts_tablet_manager.cc:1397] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.015s
I20250626 08:21:38.908205 821 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:38.908565 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:38.909554 971 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.909631 861 leader_election.cc:304] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:38.910658 1040 raft_consensus.cc:2802] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:38.911360 1016 raft_consensus.cc:357] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.912084 1016 raft_consensus.cc:383] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:21:38.911958 1040 raft_consensus.cc:695] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:38.912518 1016 raft_consensus.cc:738] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Initialized, Role: FOLLOWER
I20250626 08:21:38.913159 1016 consensus_queue.cc:260] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.913439 1040 consensus_queue.cc:237] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.916425 971 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:38.915477 1016 ts_tablet_manager.cc:1428] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250626 08:21:38.925209 719 catalog_manager.cc:5582] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 1 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:38.953472 1042 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:38.978279 1040 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:38.978832 1040 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:38.980851 1040 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:38.981074 1020 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:21:38.981698 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:38.983829 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:38.984576 971 raft_consensus.cc:2391] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e244cde94294d2887a4fa403b6b25db in current term 1: Already voted for candidate e94e795053dd4dd7aeb473e394a14400 in this term.
I20250626 08:21:38.987401 863 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db; no voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:38.988561 1040 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:38.988965 1019 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:38.996812 1019 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:21:39.005848 1040 raft_consensus.cc:2747] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:39.009972 1040 raft_consensus.cc:491] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:39.010457 1040 raft_consensus.cc:513] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.013252 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400" is_pre_election: true
I20250626 08:21:39.013873 822 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:39.015192 861 leader_election.cc:304] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.016033 1019 raft_consensus.cc:2802] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:39.016408 1019 raft_consensus.cc:491] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:21:39.016585 767 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:21:39.016765 1019 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.019505 1040 leader_election.cc:290] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.020548 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:39.021164 971 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 0.
I20250626 08:21:39.033994 1019 raft_consensus.cc:513] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.035990 1019 leader_election.cc:290] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.036955 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:39.037241 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.037664 821 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.037771 970 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.050024 821 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:39.051018 861 leader_election.cc:304] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.051757 1041 raft_consensus.cc:2802] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:39.052138 1041 raft_consensus.cc:695] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.052804 970 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 1.
I20250626 08:21:39.052806 1041 consensus_queue.cc:237] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.063091 717 catalog_manager.cc:5582] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 1 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.066525 1024 raft_consensus.cc:491] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:39.067025 1024 raft_consensus.cc:513] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.068943 1024 leader_election.cc:290] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.070519 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:39.071002 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:39.072065 970 raft_consensus.cc:2391] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e94e795053dd4dd7aeb473e394a14400 in current term 1: Already voted for candidate 5e244cde94294d2887a4fa403b6b25db in this term.
I20250626 08:21:39.072911 788 leader_election.cc:304] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e94e795053dd4dd7aeb473e394a14400; no voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef
I20250626 08:21:39.073690 1024 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.078028 1024 raft_consensus.cc:2747] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:21:39.183667 1041 consensus_queue.cc:1035] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.194940 1041 consensus_queue.cc:1035] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.225771 1024 raft_consensus.cc:491] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:21:39.226256 1024 raft_consensus.cc:513] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.227780 1024 leader_election.cc:290] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.228828 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef" is_pre_election: true
I20250626 08:21:39.228722 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db" is_pre_election: true
I20250626 08:21:39.229367 970 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:39.229467 897 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 0.
I20250626 08:21:39.230319 788 leader_election.cc:304] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.231076 1024 raft_consensus.cc:2802] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:21:39.231424 1024 raft_consensus.cc:491] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:21:39.231765 1024 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.238276 1024 raft_consensus.cc:513] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.240670 1024 leader_election.cc:290] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.241158 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:39.241341 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.241796 897 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.242043 970 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:21:39.248310 970 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:39.248314 897 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 1.
I20250626 08:21:39.249552 788 leader_election.cc:304] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.250169 1024 raft_consensus.cc:2802] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:21:39.250598 1024 raft_consensus.cc:695] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:39.251339 1024 consensus_queue.cc:237] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.254261 1058 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.259570 1058 consensus_queue.cc:1035] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.268277 718 catalog_manager.cc:5582] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.278707 1053 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.307376 1053 consensus_queue.cc:1035] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.336174 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:54082
I20250626 08:21:39.336822 896 raft_consensus.cc:604] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Received request to transfer leadership to TS e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:39.344671 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:54082
I20250626 08:21:39.345163 896 raft_consensus.cc:604] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Received request to transfer leadership to TS e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:39.361847 970 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:34296
I20250626 08:21:39.362414 970 raft_consensus.cc:604] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:39.373375 1058 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.383121 821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:44474
I20250626 08:21:39.383322 1058 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:21:39.383765 821 raft_consensus.cc:604] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:39.395825 1075 raft_consensus.cc:991] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.396248 1075 raft_consensus.cc:1079] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.401068 897 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54064
I20250626 08:21:39.401733 897 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.402132 897 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.408022 821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:44474
I20250626 08:21:39.408633 821 raft_consensus.cc:604] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:39.414291 1019 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.415410 897 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.417444 897 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.419384 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:39.419931 822 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.421208 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.421800 970 raft_consensus.cc:3053] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.422125 970 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:39.422921 970 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.424558 970 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.428134 822 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.429397 861 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.430323 1019 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.430819 1019 raft_consensus.cc:695] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.430701 897 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:54082
I20250626 08:21:39.431691 1019 consensus_queue.cc:237] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.433835 970 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.431669 897 raft_consensus.cc:604] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Received request to transfer leadership to TS bd83cd09aff945b78947107f98f030ef
I20250626 08:21:39.441152 1080 raft_consensus.cc:991] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:39.441583 1040 raft_consensus.cc:1079] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Signalling peer e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:39.443614 820 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:44448
I20250626 08:21:39.444211 820 raft_consensus.cc:491] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.444557 820 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.445827 1040 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.450683 1080 consensus_queue.cc:1035] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.444929 719 catalog_manager.cc:5582] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 1 to 2, leader changed from bd83cd09aff945b78947107f98f030ef (127.0.171.195) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 2 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.452359 1041 consensus_queue.cc:1035] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.454793 820 raft_consensus.cc:513] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.460476 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:39.461165 897 raft_consensus.cc:3053] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.461580 897 raft_consensus.cc:738] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.463021 897 consensus_queue.cc:260] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.464728 897 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.464190 820 leader_election.cc:290] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.462189 821 raft_consensus.cc:1238] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Rejecting Update request from peer 5e244cde94294d2887a4fa403b6b25db for earlier term 1. Current term is 2. Ops: []
I20250626 08:21:39.466114 969 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.477777 897 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 2.
I20250626 08:21:39.479686 786 leader_election.cc:304] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters: bd83cd09aff945b78947107f98f030ef
I20250626 08:21:39.481069 1058 raft_consensus.cc:2802] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.481529 1058 raft_consensus.cc:695] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:39.482343 1058 consensus_queue.cc:237] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.498262 719 catalog_manager.cc:5582] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 1 to 2, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.556746 1040 raft_consensus.cc:991] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:39.557178 1080 raft_consensus.cc:1079] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Signalling peer e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:39.558712 822 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:44448
I20250626 08:21:39.559171 822 raft_consensus.cc:491] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.559463 822 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.563846 822 raft_consensus.cc:513] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.565735 822 leader_election.cc:290] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.566409 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.566260 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:39.566998 971 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.567169 897 raft_consensus.cc:3053] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.567580 897 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.568225 897 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.569293 897 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.573644 897 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 2.
I20250626 08:21:39.573675 971 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 2.
I20250626 08:21:39.574834 788 leader_election.cc:304] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.575570 1058 raft_consensus.cc:2802] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.576038 1058 raft_consensus.cc:695] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:39.576815 1058 consensus_queue.cc:237] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.583556 717 catalog_manager.cc:5582] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 1 to 2, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.656770 1040 raft_consensus.cc:991] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:39.657251 1040 raft_consensus.cc:1079] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Signalling peer bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:39.658766 970 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:34268
I20250626 08:21:39.659621 970 raft_consensus.cc:491] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.660006 970 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.664834 970 raft_consensus.cc:513] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.666249 970 leader_election.cc:290] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:39.667454 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:39.667420 897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2c8fbc740e4c80bf8252ae259d5280" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:39.668107 822 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.668172 897 raft_consensus.cc:3053] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.668606 897 raft_consensus.cc:738] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.669384 897 consensus_queue.cc:260] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.670593 897 raft_consensus.cc:3058] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.674861 822 raft_consensus.cc:2466] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 2.
I20250626 08:21:39.675966 935 leader_election.cc:304] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.676764 1020 raft_consensus.cc:2802] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.676925 897 raft_consensus.cc:2466] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 2.
I20250626 08:21:39.677258 1020 raft_consensus.cc:695] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:39.678112 1020 consensus_queue.cc:237] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.684670 717 catalog_manager.cc:5582] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 1 to 2, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 2 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:39.717124 1058 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.729214 1024 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:21:39.742408 1053 raft_consensus.cc:991] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.742843 1053 raft_consensus.cc:1079] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.744766 896 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54074
I20250626 08:21:39.745327 896 raft_consensus.cc:491] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.745698 896 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.751950 896 raft_consensus.cc:513] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.753688 896 leader_election.cc:290] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.754797 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:39.755249 822 raft_consensus.cc:3053] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.755594 822 raft_consensus.cc:738] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:39.756253 822 consensus_queue.cc:260] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.757105 822 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.759249 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.759806 970 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.761844 822 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.762934 861 leader_election.cc:304] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.763713 1080 raft_consensus.cc:2802] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.764206 1080 raft_consensus.cc:695] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.765152 1080 consensus_queue.cc:237] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.767212 970 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.773659 719 catalog_manager.cc:5582] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 1 to 2, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 2 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.855593 822 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:39.856822 1080 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:39.871142 896 raft_consensus.cc:1273] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:39.872573 1058 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:39.877329 970 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:39.879277 1041 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:39.882350 971 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.894012 971 raft_consensus.cc:1273] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:39.895856 1058 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:39.913256 1053 raft_consensus.cc:991] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.913754 1053 raft_consensus.cc:1079] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:39.915403 896 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54074
I20250626 08:21:39.915931 896 raft_consensus.cc:491] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:39.916219 896 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.922442 896 raft_consensus.cc:513] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.924378 896 leader_election.cc:290] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:39.925405 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:39.926017 822 raft_consensus.cc:3053] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:39.926287 822 raft_consensus.cc:738] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:39.926187 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:39.926865 971 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.927033 822 consensus_queue.cc:260] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.928122 822 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:39.933378 822 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.934383 861 leader_election.cc:304] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:39.935019 1080 raft_consensus.cc:2802] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:39.934993 971 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:39.935601 1080 raft_consensus.cc:695] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:39.936405 1080 consensus_queue.cc:237] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:39.943080 719 catalog_manager.cc:5582] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 1 to 2, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 2 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:39.990346 896 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:39.991580 1053 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.001159 971 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.002617 1053 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.096500 896 raft_consensus.cc:1273] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.097600 1042 consensus_queue.cc:1035] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.104485 822 raft_consensus.cc:1273] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.106197 1101 consensus_queue.cc:1035] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.352869 971 raft_consensus.cc:1273] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.353986 1080 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.361279 822 raft_consensus.cc:1273] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.362869 1041 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.419421 822 raft_consensus.cc:1273] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.420487 1041 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:40.428875 971 raft_consensus.cc:1273] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:40.430645 1103 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:42.464398 687 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250626 08:21:42.465313 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:42.480705 822 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:60770
I20250626 08:21:42.481189 822 raft_consensus.cc:604] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Received request to transfer leadership to TS bd83cd09aff945b78947107f98f030ef
I20250626 08:21:42.491745 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:42.492277 896 raft_consensus.cc:604] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Received request to transfer leadership to TS bd83cd09aff945b78947107f98f030ef
I20250626 08:21:42.493340 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 08:21:42.493897 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:42.495000 1109 raft_consensus.cc:991] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:42.495471 1103 raft_consensus.cc:1079] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Signalling peer bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:42.496914 971 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:34268
I20250626 08:21:42.497470 971 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:42.497840 971 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:42.502506 971 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.503862 971 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 3 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:42.504681 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:42.504920 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:42.505280 822 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:42.505419 896 raft_consensus.cc:3053] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:42.505767 896 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:42.506544 896 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.507727 896 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:42.512053 822 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 3.
I20250626 08:21:42.513254 935 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:42.514087 1020 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:42.514324 896 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 3.
I20250626 08:21:42.515375 1020 raft_consensus.cc:695] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:42.516227 1020 consensus_queue.cc:237] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.522747 719 catalog_manager.cc:5582] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 2 to 3, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 3 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:42.663668 1058 raft_consensus.cc:991] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: : Instructing follower bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:42.664084 1116 raft_consensus.cc:1079] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Signalling peer bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:42.665441 971 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:34282
I20250626 08:21:42.665949 971 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:42.666255 971 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:42.671090 971 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.672574 971 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:42.673487 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:42.673660 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:42.673997 822 raft_consensus.cc:3053] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:42.674182 896 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:42.674297 822 raft_consensus.cc:738] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:42.674994 822 consensus_queue.cc:260] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.675886 822 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:42.678910 896 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 2.
I20250626 08:21:42.679929 935 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters:
I20250626 08:21:42.680207 822 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 2.
I20250626 08:21:42.680608 1020 raft_consensus.cc:2802] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:42.681074 1020 raft_consensus.cc:695] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:42.681761 1020 consensus_queue.cc:237] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:42.692045 719 catalog_manager.cc:5582] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 1 to 2, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 2 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:42.975708 822 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:42.976828 1020 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:42.988293 896 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:42.989606 1110 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:43.132395 822 raft_consensus.cc:1273] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:43.133545 1110 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:43.142292 896 raft_consensus.cc:1273] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:43.145169 1122 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:43.494819 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:43.499708 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:43.500104 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:44.542627 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:44.552557 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:44.749450 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:45.750759 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:45.756824 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:45.757232 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 08:21:45.763867 750 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 690
W20250626 08:21:45.764649 750 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 753
W20250626 08:21:45.765560 750 debug-util.cc:398] Leaking SignalData structure 0x7b08001e7f60 after lost signal to thread 846
W20250626 08:21:45.766435 750 debug-util.cc:398] Leaking SignalData structure 0x7b08001f00a0 after lost signal to thread 921
W20250626 08:21:45.767258 750 debug-util.cc:398] Leaking SignalData structure 0x7b08001786c0 after lost signal to thread 995
I20250626 08:21:46.758389 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:46.763077 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:46.763440 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:47.764232 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:47.769047 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:47.769407 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:48.770345 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:48.774986 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:48.775362 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:49.776129 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:49.780866 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:49.781236 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:50.782060 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:50.786670 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:50.787045 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:51.787905 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:51.793049 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:51.793455 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:52.819924 687 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250626 08:21:52.824277 821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:44474
I20250626 08:21:52.824921 821 raft_consensus.cc:604] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:52.829603 821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3501d034516444b687806dd35b56f68f"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:44474
I20250626 08:21:52.830109 821 raft_consensus.cc:604] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:52.834510 821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:44474
I20250626 08:21:52.834964 821 raft_consensus.cc:604] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:52.839368 971 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:34296
I20250626 08:21:52.839824 971 raft_consensus.cc:604] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:52.844220 971 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
mode: GRACEFUL
new_leader_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:34296
I20250626 08:21:52.844664 971 raft_consensus.cc:604] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 LEADER]: Received request to transfer leadership to TS 5e244cde94294d2887a4fa403b6b25db
I20250626 08:21:53.029595 1120 raft_consensus.cc:991] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.030098 1143 raft_consensus.cc:1079] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.031522 897 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54064
I20250626 08:21:53.031929 897 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:53.032224 897 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:53.036733 897 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.038223 897 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 4 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:53.039233 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:53.039304 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:53.039906 971 raft_consensus.cc:3053] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:21:53.040100 821 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:53.040242 971 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:53.041006 971 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.042071 971 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:53.046239 1144 raft_consensus.cc:991] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.046386 821 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 4.
I20250626 08:21:53.046934 1130 raft_consensus.cc:1079] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.047828 861 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:53.048032 971 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 4.
I20250626 08:21:53.048626 1141 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:21:53.048964 897 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54074
I20250626 08:21:53.049477 897 raft_consensus.cc:491] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:53.049746 897 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.049849 1141 raft_consensus.cc:695] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:53.050747 1141 consensus_queue.cc:237] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.054694 897 raft_consensus.cc:513] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.056489 897 leader_election.cc:290] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:53.057703 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:53.057750 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36cf77b4b10d48468e830953d3c5b450" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:53.058295 821 raft_consensus.cc:3053] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:53.058365 971 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.058559 821 raft_consensus.cc:738] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:53.059191 821 consensus_queue.cc:260] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.060356 821 raft_consensus.cc:3058] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.062813 719 catalog_manager.cc:5582] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 3 to 4, leader changed from bd83cd09aff945b78947107f98f030ef (127.0.171.195) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 4 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:53.065076 821 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.065862 971 raft_consensus.cc:2466] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.066303 861 leader_election.cc:304] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:53.067150 1141 raft_consensus.cc:2802] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:53.067682 1141 raft_consensus.cc:695] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:53.068673 1141 consensus_queue.cc:237] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.070197 1143 raft_consensus.cc:991] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.071185 1120 raft_consensus.cc:1079] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.073904 897 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54064
I20250626 08:21:53.074608 897 raft_consensus.cc:491] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:53.075076 897 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.081960 897 raft_consensus.cc:513] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.083566 897 leader_election.cc:290] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:53.084375 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:53.084410 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b4d127c6ba04cf185518373d93861d4" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:53.085058 821 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.085155 971 raft_consensus.cc:3053] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:53.085546 971 raft_consensus.cc:738] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:53.086215 971 consensus_queue.cc:260] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.087396 971 raft_consensus.cc:3058] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.089280 719 catalog_manager.cc:5582] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 2 to 3, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 3 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:53.091806 821 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.092959 861 leader_election.cc:304] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:53.093456 971 raft_consensus.cc:2466] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.093804 1141 raft_consensus.cc:2802] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:53.094311 1141 raft_consensus.cc:695] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:53.095131 1141 consensus_queue.cc:237] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.102694 719 catalog_manager.cc:5582] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 2 to 3, leader changed from bd83cd09aff945b78947107f98f030ef (127.0.171.195) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 3 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:53.133374 1144 raft_consensus.cc:991] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.133821 1144 raft_consensus.cc:1079] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.135447 897 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54074
I20250626 08:21:53.135921 897 raft_consensus.cc:491] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:53.136178 897 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.140473 1130 raft_consensus.cc:991] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: : Instructing follower 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.140908 1144 raft_consensus.cc:1079] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Signalling peer 5e244cde94294d2887a4fa403b6b25db to start an election
I20250626 08:21:53.141366 897 raft_consensus.cc:513] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.142482 896 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3501d034516444b687806dd35b56f68f"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
from {username='slave'} at 127.0.0.1:54074
I20250626 08:21:53.142783 897 leader_election.cc:290] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:53.143034 896 raft_consensus.cc:491] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:53.143435 896 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:53.143741 821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:53.144224 821 raft_consensus.cc:3053] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:53.144455 821 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:53.143741 971 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:53.145074 971 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.145421 821 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.146771 821 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:53.148829 896 raft_consensus.cc:513] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.151443 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:53.151649 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3501d034516444b687806dd35b56f68f" candidate_uuid: "5e244cde94294d2887a4fa403b6b25db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:53.151758 971 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.152020 822 raft_consensus.cc:3053] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:53.152107 970 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:53.152364 822 raft_consensus.cc:738] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:53.153136 822 consensus_queue.cc:260] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.154508 822 raft_consensus.cc:3058] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:53.154958 863 leader_election.cc:304] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters:
I20250626 08:21:53.155750 1141 raft_consensus.cc:2802] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:53.155759 896 leader_election.cc:290] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:53.156275 1141 raft_consensus.cc:695] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:53.154537 821 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 3.
I20250626 08:21:53.157135 1141 consensus_queue.cc:237] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.160313 970 raft_consensus.cc:2466] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:53.161531 863 leader_election.cc:304] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e244cde94294d2887a4fa403b6b25db, bd83cd09aff945b78947107f98f030ef; no voters:
I20250626 08:21:53.163436 1145 raft_consensus.cc:2802] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:53.161625 822 raft_consensus.cc:2466] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e244cde94294d2887a4fa403b6b25db in term 2.
I20250626 08:21:53.166478 1145 raft_consensus.cc:695] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Leader. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:53.167346 1145 consensus_queue.cc:237] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:53.175262 719 catalog_manager.cc:5582] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 2 to 3, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 3 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:53.187498 719 catalog_manager.cc:5582] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db reported cstate change: term changed from 1 to 2, leader changed from e94e795053dd4dd7aeb473e394a14400 (127.0.171.193) to 5e244cde94294d2887a4fa403b6b25db (127.0.171.194). New cstate: current_term: 2 leader_uuid: "5e244cde94294d2887a4fa403b6b25db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:53.452292 822 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:21:53.453382 1141 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:21:53.464934 970 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:21:53.466277 1155 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:21:53.485743 970 raft_consensus.cc:1273] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.486855 1155 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:53.503540 821 raft_consensus.cc:1273] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.504509 822 raft_consensus.cc:1273] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.504868 1164 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:53.505854 1155 consensus_queue.cc:1035] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:53.515072 970 raft_consensus.cc:1273] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.516235 1141 consensus_queue.cc:1035] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:53.636332 970 raft_consensus.cc:1273] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:53.637471 1155 consensus_queue.cc:1035] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:53.643625 971 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.645094 1164 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:53.645967 821 raft_consensus.cc:1273] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:53.647557 1164 consensus_queue.cc:1035] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:53.655205 822 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Refusing update from remote peer 5e244cde94294d2887a4fa403b6b25db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:53.656658 1155 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:55.882688 687 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250626 08:21:55.883522 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:55.890918 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:55.891639 896 raft_consensus.cc:604] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Received request to transfer leadership to TS e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:55.894479 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:55.894994 896 raft_consensus.cc:604] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Received request to transfer leadership to TS e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:55.897027 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:55.897442 896 raft_consensus.cc:604] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 LEADER]: Received request to transfer leadership to TS bd83cd09aff945b78947107f98f030ef
I20250626 08:21:55.899399 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:55.899808 896 raft_consensus.cc:604] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Received request to transfer leadership to TS e94e795053dd4dd7aeb473e394a14400
I20250626 08:21:55.901691 896 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
mode: GRACEFUL
new_leader_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:54530
I20250626 08:21:55.902133 896 raft_consensus.cc:604] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Received request to transfer leadership to TS bd83cd09aff945b78947107f98f030ef
I20250626 08:21:55.902978 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250626 08:21:55.903388 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:55.989430 1174 raft_consensus.cc:991] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:55.990038 1141 raft_consensus.cc:1079] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Signalling peer e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:55.991289 822 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:44448
I20250626 08:21:55.991693 822 raft_consensus.cc:491] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:55.991971 822 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:55.996912 822 raft_consensus.cc:513] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:55.998790 822 leader_election.cc:290] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:55.999800 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:56.000173 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "682f301287ec4cffbc0d74c652054cb4" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:56.000404 896 raft_consensus.cc:3053] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:21:56.000650 970 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:56.000725 896 raft_consensus.cc:738] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:56.001444 896 consensus_queue.cc:260] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.002455 896 raft_consensus.cc:3058] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:21:56.005446 970 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 2.
I20250626 08:21:56.006608 788 leader_election.cc:304] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:56.006770 896 raft_consensus.cc:2466] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 2.
I20250626 08:21:56.008234 1175 raft_consensus.cc:2802] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:21:56.009069 1175 raft_consensus.cc:695] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:56.010061 1175 consensus_queue.cc:237] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.017288 719 catalog_manager.cc:5582] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 1 to 2, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:56.272344 1174 raft_consensus.cc:991] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:56.272749 1174 raft_consensus.cc:1079] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Signalling peer bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:56.274102 970 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:34268
I20250626 08:21:56.274545 970 raft_consensus.cc:491] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:56.274791 970 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:56.279386 970 raft_consensus.cc:513] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.281162 970 leader_election.cc:290] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 4 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347)
I20250626 08:21:56.282038 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:56.282296 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da500eca704a4606932806ffc16cc0d6" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:56.282491 896 raft_consensus.cc:3053] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:21:56.282713 822 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:56.282761 896 raft_consensus.cc:738] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:56.283432 896 consensus_queue.cc:260] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.284263 896 raft_consensus.cc:3058] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:21:56.287263 822 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 4.
I20250626 08:21:56.288599 935 leader_election.cc:304] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:56.288853 896 raft_consensus.cc:2466] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 4.
I20250626 08:21:56.289331 1148 raft_consensus.cc:2802] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:21:56.290738 1148 raft_consensus.cc:695] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 4 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:56.291172 1141 raft_consensus.cc:991] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:56.291560 1174 raft_consensus.cc:1079] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 LEADER]: Signalling peer bd83cd09aff945b78947107f98f030ef to start an election
I20250626 08:21:56.291677 1148 consensus_queue.cc:237] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.292867 970 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934"
dest_uuid: "bd83cd09aff945b78947107f98f030ef"
from {username='slave'} at 127.0.0.1:34268
I20250626 08:21:56.293308 970 raft_consensus.cc:491] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:56.293637 970 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:21:56.299211 970 raft_consensus.cc:513] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.299003 719 catalog_manager.cc:5582] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 3 to 4, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 4 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:56.301209 822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
I20250626 08:21:56.301481 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89e7562cf02a4197b81a93ff05bb5934" candidate_uuid: "bd83cd09aff945b78947107f98f030ef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:56.302183 822 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:21:56.302819 896 raft_consensus.cc:3053] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:21:56.300598 970 leader_election.cc:290] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 5 election: Requested vote from peers e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347), 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447)
I20250626 08:21:56.303118 896 raft_consensus.cc:738] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:56.303855 896 consensus_queue.cc:260] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.305219 896 raft_consensus.cc:3058] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:21:56.308198 822 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 5.
I20250626 08:21:56.309072 935 leader_election.cc:304] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:56.309957 1148 raft_consensus.cc:2802] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:21:56.310499 1148 raft_consensus.cc:695] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 5 LEADER]: Becoming Leader. State: Replica: bd83cd09aff945b78947107f98f030ef, State: Running, Role: LEADER
I20250626 08:21:56.312542 896 raft_consensus.cc:2466] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd83cd09aff945b78947107f98f030ef in term 5.
I20250626 08:21:56.312256 1148 consensus_queue.cc:237] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.321693 719 catalog_manager.cc:5582] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef reported cstate change: term changed from 4 to 5, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to bd83cd09aff945b78947107f98f030ef (127.0.171.195). New cstate: current_term: 5 leader_uuid: "bd83cd09aff945b78947107f98f030ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: HEALTHY } } }
I20250626 08:21:56.356832 1141 raft_consensus.cc:991] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:56.357200 1174 raft_consensus.cc:1079] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Signalling peer e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:56.358418 822 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:44448
I20250626 08:21:56.358880 822 raft_consensus.cc:491] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:56.359135 822 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.363031 822 raft_consensus.cc:513] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.364377 822 leader_election.cc:290] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 3 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:56.365064 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:56.365360 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f17446740b4a6e83973b82c26d0b7a" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:56.365586 896 raft_consensus.cc:3053] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:56.365845 970 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.365855 896 raft_consensus.cc:738] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:56.366703 896 consensus_queue.cc:260] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.367856 896 raft_consensus.cc:3058] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.370889 970 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 3.
I20250626 08:21:56.371776 788 leader_election.cc:304] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:56.372458 896 raft_consensus.cc:2466] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 3.
I20250626 08:21:56.372606 1175 raft_consensus.cc:2802] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:56.373078 1175 raft_consensus.cc:695] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 3 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:56.374090 1175 consensus_queue.cc:237] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.380789 719 catalog_manager.cc:5582] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 2 to 3, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:56.489514 1174 raft_consensus.cc:991] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: : Instructing follower e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:56.490157 1174 raft_consensus.cc:1079] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Signalling peer e94e795053dd4dd7aeb473e394a14400 to start an election
I20250626 08:21:56.493994 822 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d"
dest_uuid: "e94e795053dd4dd7aeb473e394a14400"
from {username='slave'} at 127.0.0.1:44448
I20250626 08:21:56.494714 822 raft_consensus.cc:491] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:21:56.495309 822 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.504853 822 raft_consensus.cc:513] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.545023 822 leader_election.cc:290] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 3 election: Requested vote from peers 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447), bd83cd09aff945b78947107f98f030ef (127.0.171.195:35605)
I20250626 08:21:56.665277 897 raft_consensus.cc:1273] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:56.667071 1176 consensus_queue.cc:1035] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:21:56.669955 968 raft_consensus.cc:1273] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:21:56.671401 1176 consensus_queue.cc:1035] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:21:56.679416 970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd83cd09aff945b78947107f98f030ef"
I20250626 08:21:56.679913 970 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.687388 970 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 3.
I20250626 08:21:56.679313 896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0df58552e5234a61a6385b169f030d3d" candidate_uuid: "e94e795053dd4dd7aeb473e394a14400" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5e244cde94294d2887a4fa403b6b25db"
I20250626 08:21:56.688565 896 raft_consensus.cc:3053] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:21:56.688886 896 raft_consensus.cc:738] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e244cde94294d2887a4fa403b6b25db, State: Running, Role: LEADER
I20250626 08:21:56.688820 788 leader_election.cc:304] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd83cd09aff945b78947107f98f030ef, e94e795053dd4dd7aeb473e394a14400; no voters:
I20250626 08:21:56.689976 1190 raft_consensus.cc:2802] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:21:56.689672 896 consensus_queue.cc:260] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.690474 1190 raft_consensus.cc:695] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 3 LEADER]: Becoming Leader. State: Replica: e94e795053dd4dd7aeb473e394a14400, State: Running, Role: LEADER
I20250626 08:21:56.691072 896 raft_consensus.cc:3058] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:21:56.691438 1190 consensus_queue.cc:237] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } }
I20250626 08:21:56.697813 896 raft_consensus.cc:2466] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94e795053dd4dd7aeb473e394a14400 in term 3.
I20250626 08:21:56.712692 718 catalog_manager.cc:5582] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 reported cstate change: term changed from 2 to 3, leader changed from 5e244cde94294d2887a4fa403b6b25db (127.0.171.194) to e94e795053dd4dd7aeb473e394a14400 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "e94e795053dd4dd7aeb473e394a14400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 } health_report { overall_health: UNKNOWN } } }
I20250626 08:21:56.778438 897 raft_consensus.cc:1273] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:56.779619 1190 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:56.788151 968 raft_consensus.cc:1273] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:56.789496 1176 consensus_queue.cc:1035] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:56.811688 897 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 5 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:21:56.812703 1148 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:21:56.817759 822 raft_consensus.cc:1273] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 5 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:21:56.819372 1182 consensus_queue.cc:1035] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:21:56.825711 897 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:21:56.826975 1148 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:21:56.838337 822 raft_consensus.cc:1273] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Refusing update from remote peer bd83cd09aff945b78947107f98f030ef: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:21:56.839840 1182 consensus_queue.cc:1035] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94e795053dd4dd7aeb473e394a14400" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:21:57.030038 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:57.035318 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:57.035755 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:57.235891 897 raft_consensus.cc:1273] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:57.235894 968 raft_consensus.cc:1273] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Refusing update from remote peer e94e795053dd4dd7aeb473e394a14400: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:21:57.237409 1192 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e244cde94294d2887a4fa403b6b25db" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:21:57.238107 1190 consensus_queue.cc:1035] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd83cd09aff945b78947107f98f030ef" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:21:57.564221 918 debug-util.cc:398] Leaking SignalData structure 0x7b080012d060 after lost signal to thread 690
W20250626 08:21:57.564949 918 debug-util.cc:398] Leaking SignalData structure 0x7b080012d740 after lost signal to thread 753
W20250626 08:21:57.565744 918 debug-util.cc:398] Leaking SignalData structure 0x7b08001d3200 after lost signal to thread 846
W20250626 08:21:57.566465 918 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0a00 after lost signal to thread 921
W20250626 08:21:57.567116 918 debug-util.cc:398] Leaking SignalData structure 0x7b08001d16c0 after lost signal to thread 995
I20250626 08:21:58.036742 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:58.041711 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:58.042135 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:21:59.043099 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:21:59.048233 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:21:59.048667 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:00.049671 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:00.054358 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:00.054750 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:01.055721 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:01.061039 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:01.061534 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:02.062523 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:02.067835 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:02.068208 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:03.069262 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:03.073983 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:03.074375 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:04.075363 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:04.080425 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:04.080796 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:05.081733 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:05.086396 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:05.086769 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:06.112634 687 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250626 08:22:06.113257 687 tablet_server.cc:178] TabletServer@127.0.171.193:0 shutting down...
I20250626 08:22:06.142122 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:06.142789 687 tablet_replica.cc:331] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.143574 687 raft_consensus.cc:2241] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.144856 687 raft_consensus.cc:2270] T 0df58552e5234a61a6385b169f030d3d P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.147316 687 tablet_replica.cc:331] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.147823 687 raft_consensus.cc:2241] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.148248 687 raft_consensus.cc:2270] T 7d2c8fbc740e4c80bf8252ae259d5280 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.150125 687 tablet_replica.cc:331] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.150609 687 raft_consensus.cc:2241] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.151026 687 raft_consensus.cc:2270] T 36cf77b4b10d48468e830953d3c5b450 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.152678 687 tablet_replica.cc:331] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.153151 687 raft_consensus.cc:2241] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.154022 687 raft_consensus.cc:2270] T d6f17446740b4a6e83973b82c26d0b7a P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.155730 687 tablet_replica.cc:331] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.156316 687 raft_consensus.cc:2241] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.156757 687 raft_consensus.cc:2270] T 89e7562cf02a4197b81a93ff05bb5934 P e94e795053dd4dd7aeb473e394a14400 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.158649 687 tablet_replica.cc:331] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.159437 687 raft_consensus.cc:2241] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.160660 687 raft_consensus.cc:2270] T 682f301287ec4cffbc0d74c652054cb4 P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250626 08:22:06.161077 861 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 08:22:06.162770 687 tablet_replica.cc:331] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.163353 687 raft_consensus.cc:2241] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.163848 687 raft_consensus.cc:2270] T 1b4d127c6ba04cf185518373d93861d4 P e94e795053dd4dd7aeb473e394a14400 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250626 08:22:06.164641 861 consensus_peers.cc:487] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db -> Peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Couldn't send request to peer e94e795053dd4dd7aeb473e394a14400. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:06.166052 687 tablet_replica.cc:331] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.166564 687 raft_consensus.cc:2241] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.166945 687 raft_consensus.cc:2270] T 3501d034516444b687806dd35b56f68f P e94e795053dd4dd7aeb473e394a14400 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.168550 687 tablet_replica.cc:331] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400: stopping tablet replica
I20250626 08:22:06.168972 687 raft_consensus.cc:2241] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.169378 687 raft_consensus.cc:2270] T da500eca704a4606932806ffc16cc0d6 P e94e795053dd4dd7aeb473e394a14400 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.266305 687 tablet_server.cc:195] TabletServer@127.0.171.193:0 shutdown complete.
W20250626 08:22:06.267346 935 consensus_peers.cc:487] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef -> Peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Couldn't send request to peer e94e795053dd4dd7aeb473e394a14400. Status: Network error: Client connection negotiation failed: client connection to 127.0.171.193:36347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:06.284986 687 tablet_server.cc:178] TabletServer@127.0.171.194:0 shutting down...
I20250626 08:22:06.317747 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:06.318408 687 tablet_replica.cc:331] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.318935 687 raft_consensus.cc:2241] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.319715 687 raft_consensus.cc:2270] T 1b4d127c6ba04cf185518373d93861d4 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.321465 687 tablet_replica.cc:331] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.322031 687 raft_consensus.cc:2241] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.322415 687 raft_consensus.cc:2270] T 89e7562cf02a4197b81a93ff05bb5934 P 5e244cde94294d2887a4fa403b6b25db [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.323978 687 tablet_replica.cc:331] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.324544 687 raft_consensus.cc:2241] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.324915 687 raft_consensus.cc:2270] T 682f301287ec4cffbc0d74c652054cb4 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.326529 687 tablet_replica.cc:331] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.326951 687 raft_consensus.cc:2241] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.327678 687 raft_consensus.cc:2270] T 36cf77b4b10d48468e830953d3c5b450 P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.329131 687 tablet_replica.cc:331] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.329613 687 raft_consensus.cc:2241] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.330058 687 raft_consensus.cc:2270] T 7d2c8fbc740e4c80bf8252ae259d5280 P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.331594 687 tablet_replica.cc:331] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.332010 687 raft_consensus.cc:2241] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.332458 687 raft_consensus.cc:2270] T d6f17446740b4a6e83973b82c26d0b7a P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.334002 687 tablet_replica.cc:331] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.334416 687 raft_consensus.cc:2241] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.334832 687 raft_consensus.cc:2270] T 0df58552e5234a61a6385b169f030d3d P 5e244cde94294d2887a4fa403b6b25db [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.336413 687 tablet_replica.cc:331] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.336812 687 raft_consensus.cc:2241] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.337533 687 raft_consensus.cc:2270] T 3501d034516444b687806dd35b56f68f P 5e244cde94294d2887a4fa403b6b25db [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.339196 687 tablet_replica.cc:331] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db: stopping tablet replica
I20250626 08:22:06.339597 687 raft_consensus.cc:2241] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.339931 687 raft_consensus.cc:2270] T da500eca704a4606932806ffc16cc0d6 P 5e244cde94294d2887a4fa403b6b25db [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.396521 687 tablet_server.cc:195] TabletServer@127.0.171.194:0 shutdown complete.
I20250626 08:22:06.415522 687 tablet_server.cc:178] TabletServer@127.0.171.195:0 shutting down...
W20250626 08:22:06.429679 935 consensus_peers.cc:487] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef -> Peer 5e244cde94294d2887a4fa403b6b25db (127.0.171.194:43447): Couldn't send request to peer 5e244cde94294d2887a4fa403b6b25db. Status: Network error: Client connection negotiation failed: client connection to 127.0.171.194:43447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:06.444079 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:06.444753 687 tablet_replica.cc:331] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.445294 687 raft_consensus.cc:2241] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.445785 687 raft_consensus.cc:2270] T 0df58552e5234a61a6385b169f030d3d P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.447608 687 tablet_replica.cc:331] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.448102 687 raft_consensus.cc:2241] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.448486 687 raft_consensus.cc:2270] T 682f301287ec4cffbc0d74c652054cb4 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.450920 687 tablet_replica.cc:331] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.451493 687 raft_consensus.cc:2241] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 5 LEADER]: Raft consensus shutting down.
W20250626 08:22:06.452415 935 consensus_peers.cc:487] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef -> Peer e94e795053dd4dd7aeb473e394a14400 (127.0.171.193:36347): Couldn't send request to peer e94e795053dd4dd7aeb473e394a14400. Status: Network error: Client connection negotiation failed: client connection to 127.0.171.193:36347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:06.452493 687 raft_consensus.cc:2270] T 89e7562cf02a4197b81a93ff05bb5934 P bd83cd09aff945b78947107f98f030ef [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.454146 687 tablet_replica.cc:331] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.454582 687 raft_consensus.cc:2241] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.454953 687 raft_consensus.cc:2270] T 1b4d127c6ba04cf185518373d93861d4 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.456578 687 tablet_replica.cc:331] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.457041 687 raft_consensus.cc:2241] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.457441 687 raft_consensus.cc:2270] T d6f17446740b4a6e83973b82c26d0b7a P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.458952 687 tablet_replica.cc:331] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.459367 687 raft_consensus.cc:2241] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.460095 687 raft_consensus.cc:2270] T 7d2c8fbc740e4c80bf8252ae259d5280 P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.461560 687 tablet_replica.cc:331] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.462045 687 raft_consensus.cc:2241] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.462436 687 raft_consensus.cc:2270] T 36cf77b4b10d48468e830953d3c5b450 P bd83cd09aff945b78947107f98f030ef [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.463964 687 tablet_replica.cc:331] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.464393 687 raft_consensus.cc:2241] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:06.464815 687 raft_consensus.cc:2270] T 3501d034516444b687806dd35b56f68f P bd83cd09aff945b78947107f98f030ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.466313 687 tablet_replica.cc:331] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef: stopping tablet replica
I20250626 08:22:06.466729 687 raft_consensus.cc:2241] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 4 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.467444 687 raft_consensus.cc:2270] T da500eca704a4606932806ffc16cc0d6 P bd83cd09aff945b78947107f98f030ef [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.494359 687 tablet_server.cc:195] TabletServer@127.0.171.195:0 shutdown complete.
I20250626 08:22:06.511498 687 master.cc:561] Master@127.0.171.254:34965 shutting down...
I20250626 08:22:06.653716 687 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 1 LEADER]: Raft consensus shutting down.
I20250626 08:22:06.654377 687 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:06.654706 687 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8c26721dcad5495fb4c11659eb02d91d: stopping tablet replica
I20250626 08:22:06.689072 687 master.cc:583] Master@127.0.171.254:34965 shutdown complete.
5e244cde94294d2887a4fa403b6b25db 4
bd83cd09aff945b78947107f98f030ef 1
e94e795053dd4dd7aeb473e394a14400 4
5e244cde94294d2887a4fa403b6b25db 3
bd83cd09aff945b78947107f98f030ef 3
e94e795053dd4dd7aeb473e394a14400 3
5e244cde94294d2887a4fa403b6b25db 8
bd83cd09aff945b78947107f98f030ef 1
e94e795053dd4dd7aeb473e394a14400 0
5e244cde94294d2887a4fa403b6b25db 3
bd83cd09aff945b78947107f98f030ef 3
e94e795053dd4dd7aeb473e394a14400 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31254 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250626 08:22:06.729357 687 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.171.254:35743
I20250626 08:22:06.730496 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:06.735337 1214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:06.736392 1215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:06.736892 1217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:06.738135 687 server_base.cc:1048] running on GCE node
I20250626 08:22:06.739084 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:06.739274 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:06.739444 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926126739420 us; error 0 us; skew 500 ppm
I20250626 08:22:06.739909 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:06.742586 687 webserver.cc:469] Webserver started at http://127.0.171.254:36739/ using document root <none> and password file <none>
I20250626 08:22:06.743057 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:06.743234 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:06.743484 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:06.744578 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/master-0-root/instance:
uuid: "814ce75c1bb94487bfc9ef2b4b26ff22"
format_stamp: "Formatted at 2025-06-26 08:22:06 on dist-test-slave-660p"
I20250626 08:22:06.748906 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250626 08:22:06.752151 1222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:06.752827 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250626 08:22:06.753077 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/master-0-root
uuid: "814ce75c1bb94487bfc9ef2b4b26ff22"
format_stamp: "Formatted at 2025-06-26 08:22:06 on dist-test-slave-660p"
I20250626 08:22:06.753319 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:06.768648 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:06.769861 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:06.808188 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.254:35743
I20250626 08:22:06.808281 1273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.254:35743 every 8 connection(s)
I20250626 08:22:06.811899 1274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:06.821452 1274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22: Bootstrap starting.
I20250626 08:22:06.826057 1274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:06.830077 1274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22: No bootstrap required, opened a new log
I20250626 08:22:06.832049 1274 raft_consensus.cc:357] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER }
I20250626 08:22:06.832469 1274 raft_consensus.cc:383] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:06.832665 1274 raft_consensus.cc:738] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814ce75c1bb94487bfc9ef2b4b26ff22, State: Initialized, Role: FOLLOWER
I20250626 08:22:06.833153 1274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER }
I20250626 08:22:06.833572 1274 raft_consensus.cc:397] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:06.833765 1274 raft_consensus.cc:491] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:06.834031 1274 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:06.837997 1274 raft_consensus.cc:513] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER }
I20250626 08:22:06.838464 1274 leader_election.cc:304] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814ce75c1bb94487bfc9ef2b4b26ff22; no voters:
I20250626 08:22:06.839532 1274 leader_election.cc:290] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:22:06.839905 1277 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:06.841203 1277 raft_consensus.cc:695] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 1 LEADER]: Becoming Leader. State: Replica: 814ce75c1bb94487bfc9ef2b4b26ff22, State: Running, Role: LEADER
I20250626 08:22:06.841809 1277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER }
I20250626 08:22:06.842577 1274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:06.846913 1279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814ce75c1bb94487bfc9ef2b4b26ff22. Latest consensus state: current_term: 1 leader_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER } }
I20250626 08:22:06.847311 1278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814ce75c1bb94487bfc9ef2b4b26ff22" member_type: VOTER } }
I20250626 08:22:06.847666 1279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:06.847841 1278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:06.849701 1284 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:06.854552 1284 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:06.858033 687 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 08:22:06.863039 1284 catalog_manager.cc:1349] Generated new cluster ID: ef837310b2a04964b17e9fa34605b045
I20250626 08:22:06.863303 1284 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:06.879284 1284 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:22:06.880503 1284 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:06.897369 1284 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22: Generated new TSK 0
I20250626 08:22:06.897953 1284 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:06.924338 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:06.929982 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:06.931671 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:06.933779 687 server_base.cc:1048] running on GCE node
W20250626 08:22:06.934154 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:06.935076 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:06.935253 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:06.935384 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926126935366 us; error 0 us; skew 500 ppm
I20250626 08:22:06.935827 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:06.938130 687 webserver.cc:469] Webserver started at http://127.0.171.193:35311/ using document root <none> and password file <none>
I20250626 08:22:06.938613 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:06.938784 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:06.939021 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:06.940011 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-0-root/instance:
uuid: "642bad7878754413807793f3e5737d6a"
format_stamp: "Formatted at 2025-06-26 08:22:06 on dist-test-slave-660p"
I20250626 08:22:06.944221 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250626 08:22:06.947204 1304 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:06.947856 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 08:22:06.948112 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-0-root
uuid: "642bad7878754413807793f3e5737d6a"
format_stamp: "Formatted at 2025-06-26 08:22:06 on dist-test-slave-660p"
I20250626 08:22:06.948369 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:06.969724 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:06.970973 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:06.972422 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:06.974606 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:06.974797 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:06.975008 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:06.975170 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.013610 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.193:43537
I20250626 08:22:07.013734 1366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.193:43537 every 8 connection(s)
I20250626 08:22:07.019125 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:07.026360 1371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:07.028252 1372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:07.029980 1374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:07.030195 687 server_base.cc:1048] running on GCE node
I20250626 08:22:07.030689 1367 heartbeater.cc:344] Connected to a master server at 127.0.171.254:35743
I20250626 08:22:07.031020 1367 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:07.031710 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250626 08:22:07.031719 1367 heartbeater.cc:507] Master 127.0.171.254:35743 requested a full tablet report, sending...
W20250626 08:22:07.031968 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:07.032169 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926127032153 us; error 0 us; skew 500 ppm
I20250626 08:22:07.032696 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:07.033890 1239 ts_manager.cc:194] Registered new tserver with Master: 642bad7878754413807793f3e5737d6a (127.0.171.193:43537)
I20250626 08:22:07.035688 1239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54014
I20250626 08:22:07.036445 687 webserver.cc:469] Webserver started at http://127.0.171.194:40643/ using document root <none> and password file <none>
I20250626 08:22:07.036947 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:07.037138 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:07.037385 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:07.038552 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-1-root/instance:
uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
format_stamp: "Formatted at 2025-06-26 08:22:07 on dist-test-slave-660p"
I20250626 08:22:07.043022 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250626 08:22:07.046407 1379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.047183 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250626 08:22:07.047448 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-1-root
uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
format_stamp: "Formatted at 2025-06-26 08:22:07 on dist-test-slave-660p"
I20250626 08:22:07.047709 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:07.063756 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:07.064903 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:07.066285 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:07.068356 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:07.068532 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.068763 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:07.068905 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.106387 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.194:45853
I20250626 08:22:07.106470 1442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.194:45853 every 8 connection(s)
I20250626 08:22:07.111266 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:07.117239 1446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:07.119092 1447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:07.122018 1449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:07.122046 687 server_base.cc:1048] running on GCE node
I20250626 08:22:07.123288 1443 heartbeater.cc:344] Connected to a master server at 127.0.171.254:35743
I20250626 08:22:07.123322 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:07.123704 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:07.123739 1443 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:07.123885 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926127123869 us; error 0 us; skew 500 ppm
I20250626 08:22:07.124490 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:07.124519 1443 heartbeater.cc:507] Master 127.0.171.254:35743 requested a full tablet report, sending...
I20250626 08:22:07.126974 1239 ts_manager.cc:194] Registered new tserver with Master: 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.127955 687 webserver.cc:469] Webserver started at http://127.0.171.195:34319/ using document root <none> and password file <none>
I20250626 08:22:07.128582 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:07.128813 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:07.128768 1239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54020
I20250626 08:22:07.129220 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:07.130510 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-2-root/instance:
uuid: "5edae8a524254944a1b0d53a763a0f54"
format_stamp: "Formatted at 2025-06-26 08:22:07 on dist-test-slave-660p"
I20250626 08:22:07.134775 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250626 08:22:07.137883 1454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.138630 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250626 08:22:07.138893 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-2-root
uuid: "5edae8a524254944a1b0d53a763a0f54"
format_stamp: "Formatted at 2025-06-26 08:22:07 on dist-test-slave-660p"
I20250626 08:22:07.139156 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1750926095312382-687-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:07.149814 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:07.151046 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:07.152474 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:07.154534 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:07.154769 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.154994 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:07.155195 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:07.193197 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.195:33497
I20250626 08:22:07.193287 1516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.195:33497 every 8 connection(s)
I20250626 08:22:07.205998 1517 heartbeater.cc:344] Connected to a master server at 127.0.171.254:35743
I20250626 08:22:07.206450 1517 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:07.207162 1517 heartbeater.cc:507] Master 127.0.171.254:35743 requested a full tablet report, sending...
I20250626 08:22:07.208842 1239 ts_manager.cc:194] Registered new tserver with Master: 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.209270 687 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013031527s
I20250626 08:22:07.210155 1239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54022
I20250626 08:22:07.211642 687 test_util.cc:276] Using random seed: 1104618597
I20250626 08:22:07.233170 1239 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54030:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 08:22:07.235918 1239 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:22:07.306303 1406 tablet_service.cc:1468] Processing CreateTablet for tablet 266c5c09162147c39240daff80116ee9 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250626 08:22:07.307399 1407 tablet_service.cc:1468] Processing CreateTablet for tablet 25e1f15085a64706b7bfc464c12937a0 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250626 08:22:07.307912 1406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c5c09162147c39240daff80116ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.308678 1407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e1f15085a64706b7bfc464c12937a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.310068 1405 tablet_service.cc:1468] Processing CreateTablet for tablet efd89843bdaa4ca293eca7af3af7e163 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250626 08:22:07.311393 1405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd89843bdaa4ca293eca7af3af7e163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.311210 1481 tablet_service.cc:1468] Processing CreateTablet for tablet 266c5c09162147c39240daff80116ee9 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250626 08:22:07.312496 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c5c09162147c39240daff80116ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.314900 1404 tablet_service.cc:1468] Processing CreateTablet for tablet 19171a3757464e19b2b1229496a4ab8d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250626 08:22:07.315135 1480 tablet_service.cc:1468] Processing CreateTablet for tablet efd89843bdaa4ca293eca7af3af7e163 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250626 08:22:07.315816 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19171a3757464e19b2b1229496a4ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.316480 1480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd89843bdaa4ca293eca7af3af7e163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.319149 1479 tablet_service.cc:1468] Processing CreateTablet for tablet 19171a3757464e19b2b1229496a4ab8d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250626 08:22:07.320443 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19171a3757464e19b2b1229496a4ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.319149 1403 tablet_service.cc:1468] Processing CreateTablet for tablet 72f57604705a4929815a3c7e3cd7756d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250626 08:22:07.323338 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f57604705a4929815a3c7e3cd7756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.323104 1478 tablet_service.cc:1468] Processing CreateTablet for tablet 72f57604705a4929815a3c7e3cd7756d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250626 08:22:07.324365 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f57604705a4929815a3c7e3cd7756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.328027 1482 tablet_service.cc:1468] Processing CreateTablet for tablet 25e1f15085a64706b7bfc464c12937a0 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250626 08:22:07.329316 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e1f15085a64706b7bfc464c12937a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.334123 1532 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.341511 1532 tablet_bootstrap.cc:654] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.350443 1534 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.353406 1532 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.353998 1532 ts_tablet_manager.cc:1397] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20250626 08:22:07.357070 1534 tablet_bootstrap.cc:654] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.356882 1532 raft_consensus.cc:357] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.357710 1532 raft_consensus.cc:383] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.358094 1532 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.358978 1532 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.362038 1332 tablet_service.cc:1468] Processing CreateTablet for tablet 25e1f15085a64706b7bfc464c12937a0 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250626 08:22:07.363755 1332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e1f15085a64706b7bfc464c12937a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.368888 1534 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.369474 1534 ts_tablet_manager.cc:1397] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20250626 08:22:07.370071 1443 heartbeater.cc:499] Master 127.0.171.254:35743 was elected leader, sending a full tablet report...
I20250626 08:22:07.370791 1532 ts_tablet_manager.cc:1428] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20250626 08:22:07.371810 1532 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.378229 1532 tablet_bootstrap.cc:654] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.385727 1534 raft_consensus.cc:357] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.389317 1534 raft_consensus.cc:383] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.389869 1534 raft_consensus.cc:738] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.390457 1331 tablet_service.cc:1468] Processing CreateTablet for tablet 266c5c09162147c39240daff80116ee9 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250626 08:22:07.390710 1534 consensus_queue.cc:260] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.392168 1331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c5c09162147c39240daff80116ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.394369 1329 tablet_service.cc:1468] Processing CreateTablet for tablet 19171a3757464e19b2b1229496a4ab8d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250626 08:22:07.395712 1329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19171a3757464e19b2b1229496a4ab8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.391741 1330 tablet_service.cc:1468] Processing CreateTablet for tablet efd89843bdaa4ca293eca7af3af7e163 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250626 08:22:07.399118 1330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd89843bdaa4ca293eca7af3af7e163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.397830 1482 tablet_service.cc:1468] Processing CreateTablet for tablet 899475d7f1b345868d4ad1069d969127 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250626 08:22:07.401165 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899475d7f1b345868d4ad1069d969127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.403088 1328 tablet_service.cc:1468] Processing CreateTablet for tablet 899475d7f1b345868d4ad1069d969127 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250626 08:22:07.404309 1328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899475d7f1b345868d4ad1069d969127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.404860 1532 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.405308 1532 ts_tablet_manager.cc:1397] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.009s
I20250626 08:22:07.408010 1532 raft_consensus.cc:357] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.408695 1532 raft_consensus.cc:383] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.408967 1532 raft_consensus.cc:738] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.409721 1532 consensus_queue.cc:260] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.410763 1537 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.414860 1479 tablet_service.cc:1468] Processing CreateTablet for tablet 51d05dc235dc49e6927ccabc4dbe189e (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250626 08:22:07.416128 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d05dc235dc49e6927ccabc4dbe189e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.418188 1537 tablet_bootstrap.cc:654] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.422888 1481 tablet_service.cc:1468] Processing CreateTablet for tablet fd0ab0747ad14a24bed106da8471f77f (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250626 08:22:07.424146 1481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0ab0747ad14a24bed106da8471f77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.424052 1332 tablet_service.cc:1468] Processing CreateTablet for tablet 72f57604705a4929815a3c7e3cd7756d (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250626 08:22:07.425302 1332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f57604705a4929815a3c7e3cd7756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.436985 1517 heartbeater.cc:499] Master 127.0.171.254:35743 was elected leader, sending a full tablet report...
I20250626 08:22:07.439388 1403 tablet_service.cc:1468] Processing CreateTablet for tablet fd0ab0747ad14a24bed106da8471f77f (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250626 08:22:07.440773 1403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0ab0747ad14a24bed106da8471f77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.440466 1327 tablet_service.cc:1468] Processing CreateTablet for tablet fd0ab0747ad14a24bed106da8471f77f (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250626 08:22:07.441768 1327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0ab0747ad14a24bed106da8471f77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.444453 1478 tablet_service.cc:1468] Processing CreateTablet for tablet c3774126f95c456dbe28c6d076e13910 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250626 08:22:07.446553 1406 tablet_service.cc:1468] Processing CreateTablet for tablet 899475d7f1b345868d4ad1069d969127 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250626 08:22:07.450290 1482 tablet_service.cc:1468] Processing CreateTablet for tablet aa78ce9017fd453d949ec53b92e6f065 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250626 08:22:07.451431 1407 tablet_service.cc:1468] Processing CreateTablet for tablet 51d05dc235dc49e6927ccabc4dbe189e (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250626 08:22:07.451383 1326 tablet_service.cc:1468] Processing CreateTablet for tablet c3774126f95c456dbe28c6d076e13910 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250626 08:22:07.460893 1482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa78ce9017fd453d949ec53b92e6f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.461537 1326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3774126f95c456dbe28c6d076e13910. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.462543 1478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3774126f95c456dbe28c6d076e13910. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.452427 1404 tablet_service.cc:1468] Processing CreateTablet for tablet c3774126f95c456dbe28c6d076e13910 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250626 08:22:07.467126 1404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3774126f95c456dbe28c6d076e13910. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.456032 1325 tablet_service.cc:1468] Processing CreateTablet for tablet aa78ce9017fd453d949ec53b92e6f065 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250626 08:22:07.470034 1532 ts_tablet_manager.cc:1428] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.064s user 0.005s sys 0.004s
I20250626 08:22:07.470925 1325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa78ce9017fd453d949ec53b92e6f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.471122 1532 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.459182 1405 tablet_service.cc:1468] Processing CreateTablet for tablet aa78ce9017fd453d949ec53b92e6f065 (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250626 08:22:07.474967 1405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa78ce9017fd453d949ec53b92e6f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.475045 1534 ts_tablet_manager.cc:1428] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.092s user 0.007s sys 0.000s
I20250626 08:22:07.476171 1534 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.479321 1407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d05dc235dc49e6927ccabc4dbe189e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.482803 1406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899475d7f1b345868d4ad1069d969127. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.486609 1330 tablet_service.cc:1468] Processing CreateTablet for tablet 51d05dc235dc49e6927ccabc4dbe189e (DEFAULT_TABLE table=test-workload [id=27fec348c3dc47ed90ef0a679e3e4576]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250626 08:22:07.487563 1330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d05dc235dc49e6927ccabc4dbe189e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:07.507555 1534 tablet_bootstrap.cc:654] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.518507 1532 tablet_bootstrap.cc:654] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.533972 1537 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.534479 1537 ts_tablet_manager.cc:1397] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.124s user 0.031s sys 0.043s
I20250626 08:22:07.537140 1537 raft_consensus.cc:357] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.537871 1537 raft_consensus.cc:383] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.538687 1537 raft_consensus.cc:738] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.539484 1537 consensus_queue.cc:260] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.543679 1534 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.544278 1534 ts_tablet_manager.cc:1397] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.015s
I20250626 08:22:07.548552 1532 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.549340 1532 ts_tablet_manager.cc:1397] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.000s
I20250626 08:22:07.552881 1532 raft_consensus.cc:357] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.550983 1534 raft_consensus.cc:357] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.553854 1532 raft_consensus.cc:383] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.554443 1532 raft_consensus.cc:738] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.555696 1532 consensus_queue.cc:260] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.554172 1534 raft_consensus.cc:383] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.558215 1534 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.559010 1534 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.564955 1534 ts_tablet_manager.cc:1428] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 08:22:07.565876 1534 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.566743 1367 heartbeater.cc:499] Master 127.0.171.254:35743 was elected leader, sending a full tablet report...
I20250626 08:22:07.567505 1537 ts_tablet_manager.cc:1428] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.033s user 0.017s sys 0.007s
I20250626 08:22:07.568414 1537 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.573230 1534 tablet_bootstrap.cc:654] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.574630 1537 tablet_bootstrap.cc:654] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.576757 1532 ts_tablet_manager.cc:1428] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250626 08:22:07.583024 1532 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.583055 1537 tablet_bootstrap.cc:492] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.584169 1537 ts_tablet_manager.cc:1397] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250626 08:22:07.585839 1534 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.586324 1534 ts_tablet_manager.cc:1397] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20250626 08:22:07.586225 1537 raft_consensus.cc:357] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.587006 1537 raft_consensus.cc:383] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.587327 1537 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.588021 1537 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.589087 1534 raft_consensus.cc:357] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.589846 1534 raft_consensus.cc:383] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.590157 1534 raft_consensus.cc:738] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.590582 1537 ts_tablet_manager.cc:1428] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 08:22:07.590919 1534 consensus_queue.cc:260] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.591529 1537 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.592123 1532 tablet_bootstrap.cc:654] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.597885 1534 ts_tablet_manager.cc:1428] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20250626 08:22:07.598971 1534 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.600811 1537 tablet_bootstrap.cc:654] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.605197 1534 tablet_bootstrap.cc:654] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.606304 1532 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.606772 1532 ts_tablet_manager.cc:1397] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20250626 08:22:07.609822 1537 tablet_bootstrap.cc:492] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.610316 1537 ts_tablet_manager.cc:1397] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20250626 08:22:07.610586 1540 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.611061 1540 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.609923 1532 raft_consensus.cc:357] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.612561 1532 raft_consensus.cc:383] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.612960 1532 raft_consensus.cc:738] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.613061 1537 raft_consensus.cc:357] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.614037 1537 raft_consensus.cc:383] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.614487 1537 raft_consensus.cc:738] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.614029 1532 consensus_queue.cc:260] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.615212 1537 consensus_queue.cc:260] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.617549 1537 ts_tablet_manager.cc:1428] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250626 08:22:07.618505 1537 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.619119 1540 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.620273 1532 ts_tablet_manager.cc:1428] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250626 08:22:07.621523 1532 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.625191 1537 tablet_bootstrap.cc:654] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.629968 1532 tablet_bootstrap.cc:654] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.631935 1536 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.632551 1536 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.635572 1536 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.643190 1537 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.646624 1537 ts_tablet_manager.cc:1397] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.003s
I20250626 08:22:07.649616 1537 raft_consensus.cc:357] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.650449 1537 raft_consensus.cc:383] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.650827 1537 raft_consensus.cc:738] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.651659 1537 consensus_queue.cc:260] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.656502 1532 tablet_bootstrap.cc:492] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.657164 1532 ts_tablet_manager.cc:1397] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.000s
I20250626 08:22:07.661146 1532 raft_consensus.cc:357] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.662106 1532 raft_consensus.cc:383] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.662483 1532 raft_consensus.cc:738] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.663467 1532 consensus_queue.cc:260] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.666968 1532 ts_tablet_manager.cc:1428] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20250626 08:22:07.667887 1532 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.675351 1532 tablet_bootstrap.cc:654] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.690551 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.691407 1342 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.692683 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.693375 1417 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.694980 1543 raft_consensus.cc:491] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.695487 1543 raft_consensus.cc:513] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.696954 1537 ts_tablet_manager.cc:1428] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250626 08:22:07.697979 1537 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.705389 1537 tablet_bootstrap.cc:654] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.706190 1543 leader_election.cc:290] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.708873 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.709187 1455 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.709530 1342 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 0.
I20250626 08:22:07.710366 1540 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.710721 1540 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.711078 1540 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.710880 1380 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.712296 1536 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.712754 1536 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.713176 1536 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.715567 1532 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.716050 1532 ts_tablet_manager.cc:1397] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.004s
I20250626 08:22:07.718806 1532 raft_consensus.cc:357] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.719549 1532 raft_consensus.cc:383] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.719866 1532 raft_consensus.cc:738] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.719220 1540 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.720778 1532 consensus_queue.cc:260] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.723549 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:07.724404 1492 raft_consensus.cc:2391] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67582185b6fe4ec9ba5423577fbb6c6b in current term 1: Already voted for candidate 5edae8a524254944a1b0d53a763a0f54 in this term.
I20250626 08:22:07.726835 1534 tablet_bootstrap.cc:492] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.727068 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:07.727339 1534 ts_tablet_manager.cc:1397] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.129s user 0.037s sys 0.045s
I20250626 08:22:07.727629 1342 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.730247 1534 raft_consensus.cc:357] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.730983 1534 raft_consensus.cc:383] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.731282 1534 raft_consensus.cc:738] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.731982 1534 consensus_queue.cc:260] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.734973 1540 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.738399 1532 ts_tablet_manager.cc:1428] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.022s user 0.010s sys 0.008s
I20250626 08:22:07.733616 1536 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.740018 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.740784 1417 raft_consensus.cc:2391] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5edae8a524254944a1b0d53a763a0f54 in current term 1: Already voted for candidate 67582185b6fe4ec9ba5423577fbb6c6b in this term.
I20250626 08:22:07.742481 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:07.743227 1492 raft_consensus.cc:2391] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67582185b6fe4ec9ba5423577fbb6c6b in current term 1: Already voted for candidate 5edae8a524254944a1b0d53a763a0f54 in this term.
I20250626 08:22:07.744681 1341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:07.746315 1380 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a
I20250626 08:22:07.740865 1536 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.747686 1560 raft_consensus.cc:2747] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:07.749056 1532 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.741451 1537 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.752457 1537 ts_tablet_manager.cc:1397] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.022s
I20250626 08:22:07.758138 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19171a3757464e19b2b1229496a4ab8d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:07.759035 1492 raft_consensus.cc:2466] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 0.
I20250626 08:22:07.760747 1342 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:07.760628 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19171a3757464e19b2b1229496a4ab8d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.761564 1417 raft_consensus.cc:2466] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 0.
I20250626 08:22:07.762060 1532 tablet_bootstrap.cc:654] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.762020 1455 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters: 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:07.763269 1540 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.763725 1540 raft_consensus.cc:695] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:07.764533 1540 consensus_queue.cc:237] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.766287 1305 leader_election.cc:304] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.767477 1543 raft_consensus.cc:2802] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.767921 1543 raft_consensus.cc:491] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.768301 1543 raft_consensus.cc:3058] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.771454 1534 ts_tablet_manager.cc:1428] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.044s user 0.018s sys 0.015s
I20250626 08:22:07.772449 1534 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.778335 1537 raft_consensus.cc:357] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.779211 1537 raft_consensus.cc:383] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.779562 1537 raft_consensus.cc:738] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.780368 1543 raft_consensus.cc:513] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.780390 1537 consensus_queue.cc:260] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.783725 1543 leader_election.cc:290] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.784041 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19171a3757464e19b2b1229496a4ab8d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.784575 1417 raft_consensus.cc:3058] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.789444 1417 raft_consensus.cc:2466] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 1.
I20250626 08:22:07.790452 1305 leader_election.cc:304] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.789610 1237 catalog_manager.cc:5582] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:07.791083 1543 raft_consensus.cc:2802] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.791488 1543 raft_consensus.cc:695] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:07.792073 1543 consensus_queue.cc:237] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.794721 1534 tablet_bootstrap.cc:654] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.794935 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19171a3757464e19b2b1229496a4ab8d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:07.796025 1492 raft_consensus.cc:3058] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.803369 1532 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.804023 1532 ts_tablet_manager.cc:1397] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.032s
I20250626 08:22:07.805806 1492 raft_consensus.cc:2466] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 1.
I20250626 08:22:07.807582 1532 raft_consensus.cc:357] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.810173 1532 raft_consensus.cc:383] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.810609 1532 raft_consensus.cc:738] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.811539 1532 consensus_queue.cc:260] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.814769 1532 ts_tablet_manager.cc:1428] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20250626 08:22:07.815972 1532 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.818850 1564 raft_consensus.cc:491] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.819296 1564 raft_consensus.cc:513] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.819844 1537 ts_tablet_manager.cc:1428] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.067s user 0.007s sys 0.006s
I20250626 08:22:07.820858 1537 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.821283 1564 leader_election.cc:290] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.822067 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:07.822700 1492 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 0.
I20250626 08:22:07.823774 1306 leader_election.cc:304] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:07.823915 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.824576 1417 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 0.
I20250626 08:22:07.825527 1567 raft_consensus.cc:2802] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.825949 1567 raft_consensus.cc:491] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.826376 1532 tablet_bootstrap.cc:654] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.826311 1567 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.826563 1534 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.827049 1534 ts_tablet_manager.cc:1397] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.001s
I20250626 08:22:07.829803 1540 raft_consensus.cc:491] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.830265 1540 raft_consensus.cc:513] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.831965 1540 leader_election.cc:290] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.832763 1534 raft_consensus.cc:357] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.833129 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.833457 1534 raft_consensus.cc:383] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.833786 1564 raft_consensus.cc:491] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.833721 1417 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.834136 1534 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.834432 1564 raft_consensus.cc:513] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.835309 1455 leader_election.cc:304] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.835098 1534 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.836351 1561 raft_consensus.cc:2802] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.836369 1564 leader_election.cc:290] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.836872 1561 raft_consensus.cc:491] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.837252 1561 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.837491 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.838446 1417 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 0.
I20250626 08:22:07.840116 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.840718 1342 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.841020 1239 catalog_manager.cc:5582] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 1 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:07.842449 1305 leader_election.cc:304] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.843423 1564 raft_consensus.cc:2802] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.843770 1564 raft_consensus.cc:491] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.844089 1564 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.845779 1561 raft_consensus.cc:513] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.847791 1561 leader_election.cc:290] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.848649 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:07.848917 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.849611 1417 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:07.850056 1306 leader_election.cc:343] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.852301 1534 ts_tablet_manager.cc:1428] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
W20250626 08:22:07.852913 1287 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:07.853353 1534 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.854900 1537 tablet_bootstrap.cc:654] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.861814 1564 raft_consensus.cc:513] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.863775 1564 leader_election.cc:290] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.864923 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:07.865271 1417 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
W20250626 08:22:07.866890 1306 leader_election.cc:343] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.867198 1455 leader_election.cc:304] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:07.865667 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:07.867846 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.868183 1342 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.868443 1416 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.874641 1534 tablet_bootstrap.cc:654] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.876124 1561 raft_consensus.cc:2802] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.877146 1561 raft_consensus.cc:695] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:07.868155 1567 raft_consensus.cc:513] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:07.878046 1561 consensus_queue.cc:237] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.880872 1532 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.881006 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:07.881322 1532 ts_tablet_manager.cc:1397] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.066s user 0.026s sys 0.026s
I20250626 08:22:07.881384 1342 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:07.881598 1492 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.883827 1532 raft_consensus.cc:357] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.884284 1532 raft_consensus.cc:383] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.884533 1532 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.885057 1532 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.887151 1532 ts_tablet_manager.cc:1428] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250626 08:22:07.887881 1416 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 1.
I20250626 08:22:07.888960 1560 raft_consensus.cc:491] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.888744 1492 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 1.
I20250626 08:22:07.888171 1532 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.889551 1560 raft_consensus.cc:513] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.895007 1305 leader_election.cc:304] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:07.895228 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:07.895851 1564 raft_consensus.cc:2802] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.891325 1306 leader_election.cc:304] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:07.896342 1564 raft_consensus.cc:695] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:07.897581 1543 raft_consensus.cc:2802] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.898118 1543 raft_consensus.cc:695] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:07.898933 1543 consensus_queue.cc:237] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
W20250626 08:22:07.896494 1381 leader_election.cc:343] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.904759 1532 tablet_bootstrap.cc:654] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.904253 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.897101 1560 leader_election.cc:290] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:07.905596 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.906782 1416 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.906412 1564 consensus_queue.cc:237] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
W20250626 08:22:07.906920 1380 leader_election.cc:343] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.907435 1380 leader_election.cc:304] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a
I20250626 08:22:07.902379 1567 leader_election.cc:290] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 election: Requested vote from peers 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.910351 1560 raft_consensus.cc:2747] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:07.917771 1534 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.918308 1534 ts_tablet_manager.cc:1397] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.065s user 0.014s sys 0.001s
I20250626 08:22:07.921109 1534 raft_consensus.cc:357] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.921833 1534 raft_consensus.cc:383] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.922142 1534 raft_consensus.cc:738] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.922879 1534 consensus_queue.cc:260] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.924264 1416 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 1.
I20250626 08:22:07.925451 1534 ts_tablet_manager.cc:1428] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:22:07.930238 1532 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.930605 1532 ts_tablet_manager.cc:1397] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.010s
I20250626 08:22:07.930964 1561 raft_consensus.cc:491] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.931427 1537 tablet_bootstrap.cc:492] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:07.931769 1534 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:07.931432 1561 raft_consensus.cc:513] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.932030 1537 ts_tablet_manager.cc:1397] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.111s user 0.015s sys 0.002s
I20250626 08:22:07.933496 1561 leader_election.cc:290] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.933430 1532 raft_consensus.cc:357] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.934182 1532 raft_consensus.cc:383] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.934461 1532 raft_consensus.cc:738] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.934638 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.934971 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.935204 1532 consensus_queue.cc:260] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.935654 1416 raft_consensus.cc:2466] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
W20250626 08:22:07.935995 1455 leader_election.cc:343] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.936681 1455 leader_election.cc:304] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:07.937326 1532 ts_tablet_manager.cc:1428] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:07.938158 1561 raft_consensus.cc:2802] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.938620 1561 raft_consensus.cc:491] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.938246 1534 tablet_bootstrap.cc:654] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.938279 1537 raft_consensus.cc:357] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.939378 1537 raft_consensus.cc:383] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.939929 1537 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.939069 1561 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.940936 1532 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: Bootstrap starting.
I20250626 08:22:07.941484 1537 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.945245 1537 ts_tablet_manager.cc:1428] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250626 08:22:07.949402 1537 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:07.946857 1532 tablet_bootstrap.cc:654] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.941749 1238 catalog_manager.cc:5582] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:07.958280 1540 raft_consensus.cc:491] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:07.958801 1540 raft_consensus.cc:513] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.960673 1540 leader_election.cc:290] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.963146 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:07.963333 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:07.963798 1342 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.963861 1416 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:07.964977 1455 leader_election.cc:304] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:07.965551 1540 raft_consensus.cc:2802] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:07.965885 1540 raft_consensus.cc:491] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:07.966254 1540 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.966657 1561 raft_consensus.cc:513] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.966490 1239 catalog_manager.cc:5582] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 1 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:07.969292 1239 catalog_manager.cc:5582] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 1 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:07.971339 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
W20250626 08:22:07.972764 1455 leader_election.cc:343] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:07.974694 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.975256 1416 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.978332 1532 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: No bootstrap required, opened a new log
I20250626 08:22:07.978826 1532 ts_tablet_manager.cc:1397] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent bootstrapping tablet: real 0.038s user 0.019s sys 0.011s
I20250626 08:22:07.979835 1540 raft_consensus.cc:513] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.980695 1532 raft_consensus.cc:357] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.981324 1532 raft_consensus.cc:383] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.981601 1532 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.982316 1532 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.982546 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:07.983161 1342 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.984239 1540 leader_election.cc:290] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.983425 1534 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:07.984683 1561 leader_election.cc:290] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:07.985901 1537 tablet_bootstrap.cc:654] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:07.985601 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:07.986457 1417 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:07.988085 1532 ts_tablet_manager.cc:1428] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 08:22:07.985211 1534 ts_tablet_manager.cc:1397] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.008s
I20250626 08:22:07.991442 1534 raft_consensus.cc:357] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.992087 1534 raft_consensus.cc:383] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:07.992381 1534 raft_consensus.cc:738] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:07.993099 1534 consensus_queue.cc:260] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:07.995431 1342 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:07.996497 1455 leader_election.cc:304] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:07.997303 1561 raft_consensus.cc:2802] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:07.997767 1561 raft_consensus.cc:695] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:07.998571 1561 consensus_queue.cc:237] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.001206 1560 raft_consensus.cc:491] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.001677 1560 raft_consensus.cc:513] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.003726 1560 leader_election.cc:290] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.005225 1341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
W20250626 08:22:08.006485 1380 leader_election.cc:343] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:08.007570 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:08.008669 1417 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
W20250626 08:22:08.009232 1381 leader_election.cc:343] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:08.009469 1416 raft_consensus.cc:2466] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:08.009829 1381 leader_election.cc:304] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.011314 1534 ts_tablet_manager.cc:1428] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20250626 08:22:08.012579 1534 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:08.012351 1455 leader_election.cc:304] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.014511 1540 raft_consensus.cc:2802] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:08.013219 1560 raft_consensus.cc:2747] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:08.016631 1540 raft_consensus.cc:695] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.017381 1540 consensus_queue.cc:237] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.023764 1534 tablet_bootstrap.cc:654] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.032529 1238 catalog_manager.cc:5582] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:08.033495 1537 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:08.034164 1537 ts_tablet_manager.cc:1397] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.023s
I20250626 08:22:08.037464 1537 raft_consensus.cc:357] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.038184 1537 raft_consensus.cc:383] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.038489 1537 raft_consensus.cc:738] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.039420 1534 tablet_bootstrap.cc:492] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:08.039374 1537 consensus_queue.cc:260] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.039848 1534 ts_tablet_manager.cc:1397] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.008s
I20250626 08:22:08.041167 1560 raft_consensus.cc:491] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.041589 1560 raft_consensus.cc:513] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.043432 1560 leader_election.cc:290] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.041575 1537 ts_tablet_manager.cc:1428] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:08.044242 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:08.044538 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:08.045414 1537 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
W20250626 08:22:08.045485 1380 leader_election.cc:343] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:08.042583 1534 raft_consensus.cc:357] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.045157 1492 raft_consensus.cc:2466] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 0.
I20250626 08:22:08.046144 1534 raft_consensus.cc:383] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.048919 1534 raft_consensus.cc:738] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.049846 1381 leader_election.cc:304] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.049727 1534 consensus_queue.cc:260] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.051047 1560 raft_consensus.cc:2802] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:08.051393 1560 raft_consensus.cc:491] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:08.051716 1537 tablet_bootstrap.cc:654] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.051748 1560 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.052196 1534 ts_tablet_manager.cc:1428] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20250626 08:22:08.053155 1534 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:08.054394 1237 catalog_manager.cc:5582] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:08.058768 1534 tablet_bootstrap.cc:654] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.061420 1560 raft_consensus.cc:513] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.062846 1537 tablet_bootstrap.cc:492] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:08.063272 1537 ts_tablet_manager.cc:1397] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.010s
I20250626 08:22:08.064141 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
W20250626 08:22:08.065439 1380 leader_election.cc:343] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:08.065636 1537 raft_consensus.cc:357] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.066373 1537 raft_consensus.cc:383] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.066629 1537 raft_consensus.cc:738] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.067323 1537 consensus_queue.cc:260] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.067579 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.068063 1560 leader_election.cc:290] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.068161 1492 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.069447 1534 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:08.072022 1534 ts_tablet_manager.cc:1397] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.003s
I20250626 08:22:08.074432 1534 raft_consensus.cc:357] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.075060 1534 raft_consensus.cc:383] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.075335 1534 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.075912 1534 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.076582 1492 raft_consensus.cc:2466] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 1.
I20250626 08:22:08.077885 1534 ts_tablet_manager.cc:1428] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250626 08:22:08.077715 1381 leader_election.cc:304] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.078591 1560 raft_consensus.cc:2802] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:08.079159 1560 raft_consensus.cc:695] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.079425 1534 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: Bootstrap starting.
I20250626 08:22:08.080078 1560 consensus_queue.cc:237] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.078693 1537 ts_tablet_manager.cc:1428] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20250626 08:22:08.081542 1537 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:08.085595 1534 tablet_bootstrap.cc:654] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.088418 1537 tablet_bootstrap.cc:654] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.096093 1534 tablet_bootstrap.cc:492] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: No bootstrap required, opened a new log
I20250626 08:22:08.096748 1534 ts_tablet_manager.cc:1397] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.002s
I20250626 08:22:08.096539 1239 catalog_manager.cc:5582] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 1 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.099552 1537 tablet_bootstrap.cc:492] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:08.099551 1534 raft_consensus.cc:357] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.100095 1537 ts_tablet_manager.cc:1397] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20250626 08:22:08.100466 1534 raft_consensus.cc:383] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.100824 1534 raft_consensus.cc:738] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.101508 1534 consensus_queue.cc:260] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.102833 1537 raft_consensus.cc:357] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.103453 1537 raft_consensus.cc:383] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.103705 1537 raft_consensus.cc:738] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.104069 1534 ts_tablet_manager.cc:1428] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:22:08.104326 1537 consensus_queue.cc:260] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.106689 1537 ts_tablet_manager.cc:1428] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20250626 08:22:08.107386 1537 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: Bootstrap starting.
I20250626 08:22:08.112498 1537 tablet_bootstrap.cc:654] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:08.120070 1537 tablet_bootstrap.cc:492] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: No bootstrap required, opened a new log
I20250626 08:22:08.120409 1537 ts_tablet_manager.cc:1397] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250626 08:22:08.122064 1537 raft_consensus.cc:357] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.122519 1537 raft_consensus.cc:383] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:08.122709 1537 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Initialized, Role: FOLLOWER
I20250626 08:22:08.123168 1537 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.124622 1537 ts_tablet_manager.cc:1428] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250626 08:22:08.158042 1540 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.168805 1584 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:08.180835 1584 raft_consensus.cc:491] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.181334 1584 raft_consensus.cc:513] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.183370 1584 leader_election.cc:290] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.183868 1584 raft_consensus.cc:491] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.184293 1584 raft_consensus.cc:513] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.186022 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:08.187304 1416 raft_consensus.cc:2391] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5edae8a524254944a1b0d53a763a0f54 in current term 1: Already voted for candidate 642bad7878754413807793f3e5737d6a in this term.
I20250626 08:22:08.187165 1341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:08.187165 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:08.188670 1341 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:08.188763 1417 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:08.186208 1584 leader_election.cc:290] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.187915 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:08.189833 1455 leader_election.cc:304] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.190618 1584 raft_consensus.cc:2802] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:08.190927 1584 raft_consensus.cc:491] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:08.191246 1584 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.191174 1455 leader_election.cc:304] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54; no voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.191911 1561 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.195891 1584 raft_consensus.cc:513] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.196216 1561 raft_consensus.cc:2747] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:08.197368 1584 leader_election.cc:290] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.198124 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.198277 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.198565 1342 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.198742 1416 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.203006 1416 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:08.203058 1342 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:08.203888 1455 leader_election.cc:304] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.204471 1584 raft_consensus.cc:2802] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:08.204844 1584 raft_consensus.cc:695] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.205569 1584 consensus_queue.cc:237] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
W20250626 08:22:08.207552 1518 tablet.cc:2378] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:08.210893 1584 raft_consensus.cc:491] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.211320 1584 raft_consensus.cc:513] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.213152 1584 leader_election.cc:290] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.212886 1239 catalog_manager.cc:5582] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:08.214174 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:08.214177 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a" is_pre_election: true
I20250626 08:22:08.214941 1342 raft_consensus.cc:2466] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:08.214916 1416 raft_consensus.cc:2466] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 0.
I20250626 08:22:08.215971 1455 leader_election.cc:304] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.216555 1584 raft_consensus.cc:2802] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:08.216933 1584 raft_consensus.cc:491] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:08.217190 1584 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.218215 1543 raft_consensus.cc:491] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.218673 1543 raft_consensus.cc:513] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.220544 1543 leader_election.cc:290] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.221652 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:08.222085 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:08.222373 1492 raft_consensus.cc:2391] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 642bad7878754413807793f3e5737d6a in current term 1: Already voted for candidate 67582185b6fe4ec9ba5423577fbb6c6b in this term.
I20250626 08:22:08.223317 1584 raft_consensus.cc:513] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.223454 1306 leader_election.cc:304] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 642bad7878754413807793f3e5737d6a; no voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.224241 1543 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.225111 1584 leader_election.cc:290] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.225744 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.226138 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.226334 1342 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.226698 1416 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.229039 1543 raft_consensus.cc:2747] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:08.232901 1342 raft_consensus.cc:2466] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:08.232906 1416 raft_consensus.cc:2466] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 1.
I20250626 08:22:08.234045 1455 leader_election.cc:304] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.234632 1584 raft_consensus.cc:2802] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:08.235004 1584 raft_consensus.cc:695] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.235594 1584 consensus_queue.cc:237] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.242359 1239 catalog_manager.cc:5582] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:08.264528 1543 raft_consensus.cc:491] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:08.264948 1543 raft_consensus.cc:513] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.266569 1543 leader_election.cc:290] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.267288 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" is_pre_election: true
I20250626 08:22:08.267501 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5edae8a524254944a1b0d53a763a0f54" is_pre_election: true
I20250626 08:22:08.267899 1416 raft_consensus.cc:2391] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 642bad7878754413807793f3e5737d6a in current term 1: Already voted for candidate 5edae8a524254944a1b0d53a763a0f54 in this term.
I20250626 08:22:08.268709 1305 leader_election.cc:304] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 642bad7878754413807793f3e5737d6a; no voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.269294 1543 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:08.273548 1543 raft_consensus.cc:2747] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:08.340368 1543 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.349810 1576 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:08.380295 1543 consensus_queue.cc:1035] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.389484 1576 consensus_queue.cc:1035] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.392858 1584 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.405212 1540 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.430430 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "266c5c09162147c39240daff80116ee9"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.430950 1492 raft_consensus.cc:604] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.435420 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.435940 1492 raft_consensus.cc:604] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.442399 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.442862 1492 raft_consensus.cc:604] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.446899 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.447376 1492 raft_consensus.cc:604] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.451331 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.451783 1492 raft_consensus.cc:604] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.463011 1342 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:50590
I20250626 08:22:08.463481 1342 raft_consensus.cc:604] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.468053 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:08.468540 1492 raft_consensus.cc:604] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:08.468869 1540 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.476625 1561 raft_consensus.cc:991] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.477078 1561 raft_consensus.cc:1079] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.479018 1417 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52740
I20250626 08:22:08.480304 1417 raft_consensus.cc:491] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.480720 1417 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.487107 1416 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
mode: GRACEFUL
new_leader_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:52754
I20250626 08:22:08.487877 1416 raft_consensus.cc:604] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 LEADER]: Received request to transfer leadership to TS 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:08.486514 1417 raft_consensus.cc:513] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.488422 1561 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.491240 1417 leader_election.cc:290] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.491837 1341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.492157 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.492573 1492 raft_consensus.cc:3053] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.492828 1492 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.493481 1492 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.494359 1492 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.500133 1492 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.501132 1381 leader_election.cc:304] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.501804 1560 raft_consensus.cc:2802] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.502259 1560 raft_consensus.cc:695] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.502983 1560 consensus_queue.cc:237] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.508824 1576 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.508766 1236 catalog_manager.cc:5582] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 2 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.516378 1543 raft_consensus.cc:991] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.516825 1543 raft_consensus.cc:1079] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.518679 1416 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52748
I20250626 08:22:08.520674 1416 raft_consensus.cc:491] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.521047 1416 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.521734 1543 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.530745 1416 raft_consensus.cc:513] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.534735 1561 consensus_queue.cc:1035] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.542357 1561 raft_consensus.cc:991] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.542799 1561 raft_consensus.cc:1079] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.542888 1416 leader_election.cc:290] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.543329 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.543932 1342 raft_consensus.cc:3053] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.544234 1342 raft_consensus.cc:738] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:08.544988 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.545533 1492 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.545001 1342 consensus_queue.cc:260] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.546705 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50568
I20250626 08:22:08.547298 1340 raft_consensus.cc:491] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.547581 1340 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.547648 1342 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.553686 1584 raft_consensus.cc:991] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.554139 1584 raft_consensus.cc:1079] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.555629 1342 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.555859 1341 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50568
I20250626 08:22:08.556428 1341 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.556581 1492 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.556782 1341 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.556732 1380 leader_election.cc:304] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.557669 1560 raft_consensus.cc:2802] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.558151 1560 raft_consensus.cc:695] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.554677 1340 raft_consensus.cc:513] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.558838 1560 consensus_queue.cc:237] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.565405 1340 leader_election.cc:290] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.566037 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.566675 1417 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.568089 1238 catalog_manager.cc:5582] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 1 to 2, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 2 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.570446 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72f57604705a4929815a3c7e3cd7756d" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.571019 1492 raft_consensus.cc:3053] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.571321 1492 raft_consensus.cc:738] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.572203 1341 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.572111 1492 consensus_queue.cc:260] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.573168 1492 raft_consensus.cc:3058] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.577220 1417 raft_consensus.cc:2466] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.577446 1492 raft_consensus.cc:2466] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.578483 1305 leader_election.cc:304] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.579159 1341 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.579252 1543 raft_consensus.cc:2802] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.580197 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.580751 1543 raft_consensus.cc:695] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:08.580509 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.580200 1416 raft_consensus.cc:1238] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Rejecting Update request from peer 5edae8a524254944a1b0d53a763a0f54 for earlier term 1. Current term is 2. Ops: []
I20250626 08:22:08.581352 1417 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.581629 1543 consensus_queue.cc:237] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.580832 1492 raft_consensus.cc:3053] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.583737 1492 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.584486 1492 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.585631 1492 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.587509 1417 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.588351 1305 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.588917 1543 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.589236 1543 raft_consensus.cc:695] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:08.589826 1543 consensus_queue.cc:237] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.589783 1238 catalog_manager.cc:5582] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 2 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.591522 1492 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.601504 1238 catalog_manager.cc:5582] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 2 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.622738 1560 consensus_queue.cc:1035] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.628331 1561 consensus_queue.cc:1035] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.629159 1536 raft_consensus.cc:991] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: : Instructing follower 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:08.629804 1560 raft_consensus.cc:1079] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 LEADER]: Signalling peer 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:08.632030 1491 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:48248
I20250626 08:22:08.632596 1491 raft_consensus.cc:491] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.632952 1491 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.636498 1617 consensus_queue.cc:1035] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.638581 1540 raft_consensus.cc:991] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.639003 1540 raft_consensus.cc:1079] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.640977 1416 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52740
I20250626 08:22:08.641558 1416 raft_consensus.cc:491] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.641939 1416 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.646407 1540 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.648299 1584 consensus_queue.cc:1035] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.653497 1416 raft_consensus.cc:513] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.654845 1417 raft_consensus.cc:1238] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Rejecting Update request from peer 5edae8a524254944a1b0d53a763a0f54 for earlier term 1. Current term is 2. Ops: []
I20250626 08:22:08.656173 1491 raft_consensus.cc:513] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.658833 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.659332 1492 raft_consensus.cc:1238] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Rejecting Update request from peer 67582185b6fe4ec9ba5423577fbb6c6b for earlier term 1. Current term is 2. Ops: []
I20250626 08:22:08.658701 1584 consensus_queue.cc:1046] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.660703 1584 raft_consensus.cc:3053] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.660893 1490 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3774126f95c456dbe28c6d076e13910" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.661074 1584 raft_consensus.cc:738] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.662148 1584 consensus_queue.cc:260] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.663049 1584 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.664328 1491 leader_election.cc:290] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 2 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:08.666052 1416 leader_election.cc:290] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.668701 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.669270 1342 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.669510 1540 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.675760 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa78ce9017fd453d949ec53b92e6f065" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.671563 1617 consensus_queue.cc:1046] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.006s
I20250626 08:22:08.686053 1540 raft_consensus.cc:991] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.686326 1490 raft_consensus.cc:2466] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.686767 1561 raft_consensus.cc:1079] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:08.689201 1415 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52740
I20250626 08:22:08.689803 1415 raft_consensus.cc:491] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.690199 1415 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.694321 1381 leader_election.cc:304] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters: 642bad7878754413807793f3e5737d6a
I20250626 08:22:08.694900 1416 raft_consensus.cc:3053] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.695243 1416 raft_consensus.cc:738] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.695494 1617 raft_consensus.cc:2802] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.695957 1617 raft_consensus.cc:695] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.696033 1416 consensus_queue.cc:260] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.696817 1617 consensus_queue.cc:237] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.697324 1416 raft_consensus.cc:3058] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.705485 1342 raft_consensus.cc:2466] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 2.
I20250626 08:22:08.705506 1415 raft_consensus.cc:513] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.706804 1417 raft_consensus.cc:1238] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Rejecting Update request from peer 5edae8a524254944a1b0d53a763a0f54 for earlier term 1. Current term is 2. Ops: []
I20250626 08:22:08.707108 1455 leader_election.cc:304] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:08.708388 1584 raft_consensus.cc:2802] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.708838 1540 consensus_queue.cc:1046] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:08.709653 1561 raft_consensus.cc:3053] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.710234 1561 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.710996 1561 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.708446 1238 catalog_manager.cc:5582] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 2 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.711952 1561 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.713946 1584 raft_consensus.cc:695] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.714669 1490 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.714848 1584 consensus_queue.cc:237] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.715305 1415 leader_election.cc:290] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.711905 1342 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:08.720615 1342 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.722139 1416 raft_consensus.cc:2466] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 2.
W20250626 08:22:08.723831 1536 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250626 08:22:08.723248 1236 catalog_manager.cc:5582] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 1 to 2, leader changed from 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194) to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:08.727730 1342 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.728073 1490 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 2.
I20250626 08:22:08.728909 1380 leader_election.cc:304] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.729687 1536 raft_consensus.cc:2802] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.730237 1536 raft_consensus.cc:695] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:08.731320 1536 consensus_queue.cc:237] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.739432 1237 catalog_manager.cc:5582] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 2 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:08.886307 1490 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:08.887405 1536 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:08.894181 1342 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.900617 1342 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:08.901619 1617 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:08.951866 1540 raft_consensus.cc:991] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.952231 1584 raft_consensus.cc:1079] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:08.953413 1342 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "266c5c09162147c39240daff80116ee9"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50568
I20250626 08:22:08.953816 1342 raft_consensus.cc:491] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:08.954070 1342 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.958456 1342 raft_consensus.cc:513] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.959966 1342 leader_election.cc:290] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:08.960692 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:08.961052 1490 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:08.961275 1416 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.961602 1490 raft_consensus.cc:3053] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:08.961946 1490 raft_consensus.cc:738] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:08.962782 1490 consensus_queue.cc:260] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.963953 1490 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:08.968093 1416 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.969281 1305 leader_election.cc:304] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:08.970108 1543 raft_consensus.cc:2802] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:08.970482 1543 raft_consensus.cc:695] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:08.970496 1490 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 2.
I20250626 08:22:08.971361 1543 consensus_queue.cc:237] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:08.977788 1237 catalog_manager.cc:5582] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 1 to 2, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 2 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:09.027107 1342 raft_consensus.cc:1273] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.028209 1617 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.035802 1490 raft_consensus.cc:1273] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.037048 1536 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.092518 1416 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.093590 1543 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.100129 1490 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.101513 1567 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.124182 1342 raft_consensus.cc:1273] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.125200 1584 consensus_queue.cc:1035] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.132475 1416 raft_consensus.cc:1273] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.134421 1584 consensus_queue.cc:1035] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.154752 1490 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.155822 1536 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.161137 1416 raft_consensus.cc:1273] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.162843 1567 consensus_queue.cc:1035] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:09.163759 1342 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.165248 1623 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:09.179778 1490 raft_consensus.cc:1273] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.181732 1567 consensus_queue.cc:1035] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.190414 1490 raft_consensus.cc:1273] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.191442 1536 consensus_queue.cc:1035] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.198319 1342 raft_consensus.cc:3058] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:09.204783 1342 raft_consensus.cc:1273] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.205873 1536 consensus_queue.cc:1035] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.389030 1491 raft_consensus.cc:1273] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.392729 1564 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:09.400743 1416 raft_consensus.cc:1273] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:09.403142 1567 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:22:10.323805 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9220 after lost signal to thread 690
W20250626 08:22:10.324565 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9900 after lost signal to thread 1273
W20250626 08:22:10.325342 1513 debug-util.cc:398] Leaking SignalData structure 0x7b0800197d80 after lost signal to thread 1366
W20250626 08:22:10.326097 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a27a0 after lost signal to thread 1442
W20250626 08:22:10.326767 1513 debug-util.cc:398] Leaking SignalData structure 0x7b0800129ea0 after lost signal to thread 1516
I20250626 08:22:11.699438 687 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250626 08:22:11.700053 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:11.723021 1342 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:11.723489 1342 raft_consensus.cc:604] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Received request to transfer leadership to TS 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:11.725845 1340 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:11.726451 1340 raft_consensus.cc:604] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Received request to transfer leadership to TS 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:11.727403 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 08:22:11.727833 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:11.910140 1631 raft_consensus.cc:991] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: : Instructing follower 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:11.910676 1631 raft_consensus.cc:1079] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Signalling peer 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:11.912299 1492 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:48256
I20250626 08:22:11.912911 1492 raft_consensus.cc:491] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:11.913368 1492 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:11.921118 1492 raft_consensus.cc:513] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:11.923172 1492 leader_election.cc:290] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 2 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 642bad7878754413807793f3e5737d6a (127.0.171.193:43537)
I20250626 08:22:11.924031 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:11.924309 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:11.924626 1416 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:11.924933 1340 raft_consensus.cc:3053] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:22:11.925269 1340 raft_consensus.cc:738] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:11.926054 1340 consensus_queue.cc:260] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:11.926957 1340 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:11.931782 1416 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 2.
I20250626 08:22:11.933109 1340 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 2.
I20250626 08:22:11.933159 1455 leader_election.cc:304] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:11.934105 1626 raft_consensus.cc:2802] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:11.935186 1626 raft_consensus.cc:695] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:11.935923 1626 consensus_queue.cc:237] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:11.943920 1238 catalog_manager.cc:5582] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 1 to 2, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:11.965204 1627 raft_consensus.cc:991] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: : Instructing follower 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:11.965622 1627 raft_consensus.cc:1079] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Signalling peer 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:11.966969 1492 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:48256
I20250626 08:22:11.967404 1492 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:11.967655 1492 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:11.973028 1492 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:11.974717 1492 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 3 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:11.975270 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:11.975911 1340 raft_consensus.cc:3053] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:11.975972 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:11.976327 1340 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:11.976563 1416 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:11.977181 1340 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:11.978528 1340 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:11.980715 1416 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 3.
I20250626 08:22:11.982631 1455 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:11.983502 1626 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:11.983970 1626 raft_consensus.cc:695] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:11.984760 1626 consensus_queue.cc:237] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:11.986001 1340 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 3.
I20250626 08:22:11.992786 1238 catalog_manager.cc:5582] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 2 to 3, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:12.483777 1340 raft_consensus.cc:1273] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:12.485208 1626 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:12.514413 1415 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:12.515969 1640 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:12.517176 1416 raft_consensus.cc:1273] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:22:12.518683 1626 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:12.541973 1340 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:12.551921 1640 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:13.185879 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:13.200870 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:13.201395 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 08:22:14.166203 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4840 after lost signal to thread 690
W20250626 08:22:14.166900 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08001281a0 after lost signal to thread 1273
W20250626 08:22:14.167647 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a98c0 after lost signal to thread 1366
W20250626 08:22:14.168316 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000e46a0 after lost signal to thread 1442
W20250626 08:22:14.169016 1513 debug-util.cc:398] Leaking SignalData structure 0x7b08000b8c20 after lost signal to thread 1516
I20250626 08:22:14.202785 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:14.210229 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:14.210598 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:15.211440 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:15.219913 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:15.220458 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:16.222050 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:16.231755 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:16.232184 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:17.233228 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:17.238723 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:17.239128 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:18.240000 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:18.245069 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:18.245482 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:19.246395 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:19.251811 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:19.252202 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:20.253033 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:20.258136 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:20.258550 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:21.259426 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:21.264600 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:21.265039 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:22.296619 687 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250626 08:22:22.301520 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:22.302421 1492 raft_consensus.cc:604] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:22.308949 1492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:48266
I20250626 08:22:22.309331 1492 raft_consensus.cc:604] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:22.317891 1416 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:52754
I20250626 08:22:22.318379 1416 raft_consensus.cc:604] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:22.320029 1661 raft_consensus.cc:991] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.320418 1661 raft_consensus.cc:1079] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.321857 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50584
I20250626 08:22:22.322376 1340 raft_consensus.cc:491] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:22.322710 1340 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.323062 1416 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:52754
I20250626 08:22:22.323513 1416 raft_consensus.cc:604] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:22.328063 1416 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
mode: GRACEFUL
new_leader_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:52754
I20250626 08:22:22.328485 1416 raft_consensus.cc:604] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Received request to transfer leadership to TS 642bad7878754413807793f3e5737d6a
I20250626 08:22:22.328536 1340 raft_consensus.cc:513] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.330497 1340 leader_election.cc:290] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:22.331524 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:22.331836 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:22.332201 1416 raft_consensus.cc:3053] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:22.332306 1492 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.332546 1416 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:22.333323 1416 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.334610 1416 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.338999 1492 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.340193 1306 leader_election.cc:304] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:22.341130 1663 raft_consensus.cc:2802] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:22.341229 1416 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.341516 1663 raft_consensus.cc:695] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:22.342458 1663 consensus_queue.cc:237] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.352530 1237 catalog_manager.cc:5582] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 2 to 3, leader changed from 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 3 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:22.420536 1661 raft_consensus.cc:991] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.421084 1664 raft_consensus.cc:1079] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.422384 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50584
I20250626 08:22:22.422816 1340 raft_consensus.cc:491] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:22.423071 1340 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.427181 1340 raft_consensus.cc:513] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.428480 1340 leader_election.cc:290] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:22.429255 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:22.429443 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:22.429754 1416 raft_consensus.cc:3053] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:22.429863 1492 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.430078 1416 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:22.430778 1416 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.431581 1416 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.433883 1492 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.434737 1306 leader_election.cc:304] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:22.435369 1663 raft_consensus.cc:2802] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:22.435823 1663 raft_consensus.cc:695] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:22.436048 1416 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.436841 1663 consensus_queue.cc:237] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.443861 1237 catalog_manager.cc:5582] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 2 to 3, leader changed from 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 3 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:22.472414 1661 raft_consensus.cc:991] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.472978 1664 raft_consensus.cc:1079] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.474488 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50584
I20250626 08:22:22.474907 1340 raft_consensus.cc:491] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:22.475164 1340 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.479540 1340 raft_consensus.cc:513] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.480803 1340 leader_election.cc:290] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:22.481709 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:22.481741 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd0ab0747ad14a24bed106da8471f77f" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:22.482251 1416 raft_consensus.cc:3053] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:22.482299 1492 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.482535 1416 raft_consensus.cc:738] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:22.483127 1416 consensus_queue.cc:260] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.483932 1416 raft_consensus.cc:3058] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.486404 1492 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.487268 1306 leader_election.cc:304] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 642bad7878754413807793f3e5737d6a; no voters:
I20250626 08:22:22.488071 1663 raft_consensus.cc:2802] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:22.488281 1416 raft_consensus.cc:2466] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.488512 1663 raft_consensus.cc:695] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:22.489583 1663 consensus_queue.cc:237] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.496661 1237 catalog_manager.cc:5582] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 2 to 3, leader changed from 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 3 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:22.734489 1652 raft_consensus.cc:991] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.734884 1652 raft_consensus.cc:1079] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.736034 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50568
I20250626 08:22:22.736403 1340 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:22.736644 1340 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:22.740770 1340 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.742578 1340 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 4 election: Requested vote from peers 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:22.743301 1416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:22.743546 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:22.743781 1416 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:22.744028 1492 raft_consensus.cc:3053] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:22:22.744284 1492 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:22.744917 1492 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.745805 1492 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:22.748111 1416 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 4.
I20250626 08:22:22.749080 1305 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:22.749781 1663 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:22:22.750063 1492 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 4.
I20250626 08:22:22.750237 1663 raft_consensus.cc:695] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:22.751053 1663 consensus_queue.cc:237] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:22.759179 1237 catalog_manager.cc:5582] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 3 to 4, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 4 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:22.833034 1492 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.834115 1663 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:22.840766 1672 raft_consensus.cc:991] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: : Instructing follower 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.841360 1652 raft_consensus.cc:1079] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Signalling peer 642bad7878754413807793f3e5737d6a to start an election
I20250626 08:22:22.842844 1340 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0"
dest_uuid: "642bad7878754413807793f3e5737d6a"
from {username='slave'} at 127.0.0.1:50568
I20250626 08:22:22.843364 1340 raft_consensus.cc:491] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:22.843693 1340 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.844995 1416 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.846565 1662 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:22.850771 1340 raft_consensus.cc:513] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:22.853255 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:22.853762 1492 raft_consensus.cc:3053] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:22.854069 1492 raft_consensus.cc:738] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:22.854270 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25e1f15085a64706b7bfc464c12937a0" candidate_uuid: "642bad7878754413807793f3e5737d6a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:22.854862 1417 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.854772 1492 consensus_queue.cc:260] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:22.855857 1492 raft_consensus.cc:3058] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:22.857962 1340 leader_election.cc:290] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Requested vote from peers 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:22.861503 1417 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.861858 1492 raft_consensus.cc:2466] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642bad7878754413807793f3e5737d6a in term 3.
I20250626 08:22:22.862498 1305 leader_election.cc:304] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bad7878754413807793f3e5737d6a, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:22.863226 1663 raft_consensus.cc:2802] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:22.863723 1663 raft_consensus.cc:695] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Leader. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:22.864437 1663 consensus_queue.cc:237] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } }
I20250626 08:22:22.871248 1238 catalog_manager.cc:5582] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a reported cstate change: term changed from 2 to 3, leader changed from 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195) to 642bad7878754413807793f3e5737d6a (127.0.171.193). New cstate: current_term: 3 leader_uuid: "642bad7878754413807793f3e5737d6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:22.881160 1417 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.882371 1662 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:22.890148 1492 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.891711 1663 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:22.946031 1417 raft_consensus.cc:1273] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.947160 1663 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:22.956733 1492 raft_consensus.cc:1273] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:22.958091 1663 consensus_queue.cc:1035] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:23.133987 1492 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:23.135381 1663 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:23.144867 1417 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:23.146725 1663 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:23.355773 1417 raft_consensus.cc:1273] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:23.356956 1686 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:23.366218 1492 raft_consensus.cc:1273] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Refusing update from remote peer 642bad7878754413807793f3e5737d6a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:23.367750 1663 consensus_queue.cc:1035] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:25.366879 687 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250626 08:22:25.367516 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:25.374660 1340 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "266c5c09162147c39240daff80116ee9"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:25.375144 1340 raft_consensus.cc:604] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Received request to transfer leadership to TS 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:25.377555 1340 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:25.378037 1340 raft_consensus.cc:604] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:25.380118 1340 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:25.380514 1340 raft_consensus.cc:604] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Received request to transfer leadership to TS 5edae8a524254944a1b0d53a763a0f54
I20250626 08:22:25.382469 1340 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "642bad7878754413807793f3e5737d6a"
mode: GRACEFUL
new_leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:59930
I20250626 08:22:25.382836 1340 raft_consensus.cc:604] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 LEADER]: Received request to transfer leadership to TS 67582185b6fe4ec9ba5423577fbb6c6b
I20250626 08:22:25.383661 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 08:22:25.384070 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:25.430989 1663 raft_consensus.cc:991] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: : Instructing follower 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:25.431416 1692 raft_consensus.cc:1079] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Signalling peer 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:25.432700 1492 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "266c5c09162147c39240daff80116ee9"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:48256
I20250626 08:22:25.433092 1492 raft_consensus.cc:491] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:25.433336 1492 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:25.437531 1492 raft_consensus.cc:513] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.439118 1492 leader_election.cc:290] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 3 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:25.439950 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:25.440126 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266c5c09162147c39240daff80116ee9" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:25.440517 1340 raft_consensus.cc:3053] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:22:25.440604 1417 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:25.440845 1340 raft_consensus.cc:738] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:25.441579 1340 consensus_queue.cc:260] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.442757 1340 raft_consensus.cc:3058] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:22:25.447206 1417 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 3.
I20250626 08:22:25.448441 1455 leader_election.cc:304] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:25.448827 1340 raft_consensus.cc:2466] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 3.
I20250626 08:22:25.449054 1652 raft_consensus.cc:2802] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:22:25.450176 1652 raft_consensus.cc:695] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:25.450887 1652 consensus_queue.cc:237] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.456876 1238 catalog_manager.cc:5582] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 2 to 3, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:25.563737 1692 raft_consensus.cc:991] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:25.564159 1663 raft_consensus.cc:1079] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:25.565376 1417 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52748
I20250626 08:22:25.565793 1417 raft_consensus.cc:491] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:25.566066 1417 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:22:25.570149 1417 raft_consensus.cc:513] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.571717 1417 leader_election.cc:290] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 5 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:25.572515 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:25.572640 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efd89843bdaa4ca293eca7af3af7e163" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:25.573101 1340 raft_consensus.cc:3053] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:22:25.573176 1492 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:22:25.573374 1340 raft_consensus.cc:738] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:25.574057 1340 consensus_queue.cc:260] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.574852 1340 raft_consensus.cc:3058] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:22:25.577399 1492 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 5.
I20250626 08:22:25.578518 1381 leader_election.cc:304] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:25.579391 1690 raft_consensus.cc:2802] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:22:25.580156 1340 raft_consensus.cc:2466] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 5.
I20250626 08:22:25.580740 1690 raft_consensus.cc:695] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 5 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:25.581617 1690 consensus_queue.cc:237] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.588127 1238 catalog_manager.cc:5582] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 4 to 5, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 5 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:25.684780 1692 raft_consensus.cc:991] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: : Instructing follower 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:25.685489 1663 raft_consensus.cc:1079] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Signalling peer 5edae8a524254944a1b0d53a763a0f54 to start an election
I20250626 08:22:25.686940 1492 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "899475d7f1b345868d4ad1069d969127"
dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
from {username='slave'} at 127.0.0.1:48256
I20250626 08:22:25.687469 1492 raft_consensus.cc:491] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:25.687759 1492 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.693507 1492 raft_consensus.cc:513] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.695150 1492 leader_election.cc:290] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 4 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853)
I20250626 08:22:25.696734 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:25.697023 1417 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "899475d7f1b345868d4ad1069d969127" candidate_uuid: "5edae8a524254944a1b0d53a763a0f54" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
I20250626 08:22:25.697337 1340 raft_consensus.cc:3053] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:22:25.697610 1417 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.697652 1340 raft_consensus.cc:738] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:25.698593 1340 consensus_queue.cc:260] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.700381 1340 raft_consensus.cc:3058] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.704362 1417 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 4.
I20250626 08:22:25.705384 1455 leader_election.cc:304] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:25.706327 1652 raft_consensus.cc:2802] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:22:25.706674 1652 raft_consensus.cc:695] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 4 LEADER]: Becoming Leader. State: Replica: 5edae8a524254944a1b0d53a763a0f54, State: Running, Role: LEADER
I20250626 08:22:25.707677 1652 consensus_queue.cc:237] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.706697 1340 raft_consensus.cc:2466] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5edae8a524254944a1b0d53a763a0f54 in term 4.
I20250626 08:22:25.716588 1238 catalog_manager.cc:5582] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 reported cstate change: term changed from 3 to 4, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195). New cstate: current_term: 4 leader_uuid: "5edae8a524254944a1b0d53a763a0f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:25.762123 1692 raft_consensus.cc:991] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: : Instructing follower 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:25.762578 1663 raft_consensus.cc:1079] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Signalling peer 67582185b6fe4ec9ba5423577fbb6c6b to start an election
I20250626 08:22:25.763782 1417 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e"
dest_uuid: "67582185b6fe4ec9ba5423577fbb6c6b"
from {username='slave'} at 127.0.0.1:52748
I20250626 08:22:25.764170 1417 raft_consensus.cc:491] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:22:25.764436 1417 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.768676 1417 raft_consensus.cc:513] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.770834 1417 leader_election.cc:290] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 4 election: Requested vote from peers 642bad7878754413807793f3e5737d6a (127.0.171.193:43537), 5edae8a524254944a1b0d53a763a0f54 (127.0.171.195:33497)
I20250626 08:22:25.771452 1340 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "642bad7878754413807793f3e5737d6a"
I20250626 08:22:25.771533 1492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d05dc235dc49e6927ccabc4dbe189e" candidate_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5edae8a524254944a1b0d53a763a0f54"
I20250626 08:22:25.772120 1340 raft_consensus.cc:3053] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:22:25.772329 1492 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.772390 1340 raft_consensus.cc:738] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 642bad7878754413807793f3e5737d6a, State: Running, Role: LEADER
I20250626 08:22:25.773192 1340 consensus_queue.cc:260] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.774338 1340 raft_consensus.cc:3058] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:22:25.776741 1492 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 4.
I20250626 08:22:25.777660 1381 leader_election.cc:304] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5edae8a524254944a1b0d53a763a0f54, 67582185b6fe4ec9ba5423577fbb6c6b; no voters:
I20250626 08:22:25.778458 1690 raft_consensus.cc:2802] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:22:25.778813 1690 raft_consensus.cc:695] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 LEADER]: Becoming Leader. State: Replica: 67582185b6fe4ec9ba5423577fbb6c6b, State: Running, Role: LEADER
I20250626 08:22:25.778718 1340 raft_consensus.cc:2466] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67582185b6fe4ec9ba5423577fbb6c6b in term 4.
I20250626 08:22:25.780012 1690 consensus_queue.cc:237] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } }
I20250626 08:22:25.787060 1238 catalog_manager.cc:5582] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b reported cstate change: term changed from 3 to 4, leader changed from 642bad7878754413807793f3e5737d6a (127.0.171.193) to 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194). New cstate: current_term: 4 leader_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:25.921195 1340 raft_consensus.cc:1273] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:25.922248 1652 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:25.932571 1416 raft_consensus.cc:1273] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:22:25.933761 1652 consensus_queue.cc:1035] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:25.978802 1340 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 5 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:22:25.979907 1690 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:22:25.989439 1492 raft_consensus.cc:1273] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 5 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:22:25.990643 1690 consensus_queue.cc:1035] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:22:26.107062 1340 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:26.108136 1652 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:26.116593 1416 raft_consensus.cc:1273] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Refusing update from remote peer 5edae8a524254944a1b0d53a763a0f54: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:26.118433 1652 consensus_queue.cc:1035] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67582185b6fe4ec9ba5423577fbb6c6b" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:26.237092 1340 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:26.238193 1690 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bad7878754413807793f3e5737d6a" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:26.248201 1492 raft_consensus.cc:1273] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Refusing update from remote peer 67582185b6fe4ec9ba5423577fbb6c6b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:22:26.249496 1715 consensus_queue.cc:1035] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5edae8a524254944a1b0d53a763a0f54" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:22:26.385147 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:26.390569 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:26.390993 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:27.391976 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:27.397295 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:27.397717 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:28.398545 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:28.403645 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:28.404091 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:29.404893 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:29.410126 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:29.410511 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:30.411397 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:30.416899 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:30.417351 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:31.418243 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:31.423558 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:31.423980 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:32.424746 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:32.429868 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:32.430279 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:33.431174 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:33.436707 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:33.437153 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:34.438086 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:22:34.444039 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 08:22:34.444465 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:22:35.475872 687 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250626 08:22:35.476500 687 tablet_server.cc:178] TabletServer@127.0.171.193:0 shutting down...
I20250626 08:22:35.505246 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:35.505987 687 tablet_replica.cc:331] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.507014 687 raft_consensus.cc:2241] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.507695 687 raft_consensus.cc:2270] T aa78ce9017fd453d949ec53b92e6f065 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.509467 687 tablet_replica.cc:331] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.510059 687 raft_consensus.cc:2241] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.510396 687 raft_consensus.cc:2270] T 51d05dc235dc49e6927ccabc4dbe189e P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.511991 687 tablet_replica.cc:331] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.512435 687 raft_consensus.cc:2241] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.512779 687 raft_consensus.cc:2270] T 899475d7f1b345868d4ad1069d969127 P 642bad7878754413807793f3e5737d6a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.514324 687 tablet_replica.cc:331] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.514782 687 raft_consensus.cc:2241] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.515489 687 raft_consensus.cc:2270] T 72f57604705a4929815a3c7e3cd7756d P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.517134 687 tablet_replica.cc:331] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.517611 687 raft_consensus.cc:2241] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.518020 687 raft_consensus.cc:2270] T c3774126f95c456dbe28c6d076e13910 P 642bad7878754413807793f3e5737d6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.519594 687 tablet_replica.cc:331] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.520087 687 raft_consensus.cc:2241] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.520787 687 raft_consensus.cc:2270] T fd0ab0747ad14a24bed106da8471f77f P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.522668 687 tablet_replica.cc:331] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.523133 687 raft_consensus.cc:2241] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.523836 687 raft_consensus.cc:2270] T 25e1f15085a64706b7bfc464c12937a0 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.525656 687 tablet_replica.cc:331] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.526163 687 raft_consensus.cc:2241] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.526532 687 raft_consensus.cc:2270] T 266c5c09162147c39240daff80116ee9 P 642bad7878754413807793f3e5737d6a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.527992 687 tablet_replica.cc:331] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.528424 687 raft_consensus.cc:2241] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.528796 687 raft_consensus.cc:2270] T efd89843bdaa4ca293eca7af3af7e163 P 642bad7878754413807793f3e5737d6a [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.530297 687 tablet_replica.cc:331] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a: stopping tablet replica
I20250626 08:22:35.530699 687 raft_consensus.cc:2241] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 1 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.531333 687 raft_consensus.cc:2270] T 19171a3757464e19b2b1229496a4ab8d P 642bad7878754413807793f3e5737d6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.560444 687 tablet_server.cc:195] TabletServer@127.0.171.193:0 shutdown complete.
I20250626 08:22:35.579655 687 tablet_server.cc:178] TabletServer@127.0.171.194:0 shutting down...
W20250626 08:22:35.584096 1455 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20250626 08:22:35.588940 1455 consensus_peers.cc:487] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 -> Peer 67582185b6fe4ec9ba5423577fbb6c6b (127.0.171.194:45853): Couldn't send request to peer 67582185b6fe4ec9ba5423577fbb6c6b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:35.608852 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:35.609509 687 tablet_replica.cc:331] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.610284 687 raft_consensus.cc:2241] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.610967 687 raft_consensus.cc:2270] T 51d05dc235dc49e6927ccabc4dbe189e P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.612571 687 tablet_replica.cc:331] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.613035 687 raft_consensus.cc:2241] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.613386 687 raft_consensus.cc:2270] T 899475d7f1b345868d4ad1069d969127 P 67582185b6fe4ec9ba5423577fbb6c6b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.615057 687 tablet_replica.cc:331] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.615695 687 raft_consensus.cc:2241] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.616240 687 raft_consensus.cc:2270] T aa78ce9017fd453d949ec53b92e6f065 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.618482 687 tablet_replica.cc:331] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.619189 687 raft_consensus.cc:2241] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.619546 687 raft_consensus.cc:2270] T 266c5c09162147c39240daff80116ee9 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus is shut down!
W20250626 08:22:35.619726 1455 consensus_peers.cc:487] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 -> Peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Couldn't send request to peer 642bad7878754413807793f3e5737d6a. Status: Network error: Client connection negotiation failed: client connection to 127.0.171.193:43537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:35.621510 687 tablet_replica.cc:331] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.622164 687 raft_consensus.cc:2241] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.622596 687 raft_consensus.cc:2270] T 25e1f15085a64706b7bfc464c12937a0 P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.624311 687 tablet_replica.cc:331] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.624794 687 raft_consensus.cc:2241] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 5 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.625435 687 raft_consensus.cc:2270] T efd89843bdaa4ca293eca7af3af7e163 P 67582185b6fe4ec9ba5423577fbb6c6b [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.627068 687 tablet_replica.cc:331] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.627516 687 raft_consensus.cc:2241] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.627827 687 raft_consensus.cc:2270] T 19171a3757464e19b2b1229496a4ab8d P 67582185b6fe4ec9ba5423577fbb6c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.629391 687 tablet_replica.cc:331] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.629813 687 raft_consensus.cc:2241] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.630538 687 raft_consensus.cc:2270] T c3774126f95c456dbe28c6d076e13910 P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.632078 687 tablet_replica.cc:331] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.632498 687 raft_consensus.cc:2241] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.632834 687 raft_consensus.cc:2270] T fd0ab0747ad14a24bed106da8471f77f P 67582185b6fe4ec9ba5423577fbb6c6b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.634370 687 tablet_replica.cc:331] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b: stopping tablet replica
I20250626 08:22:35.634738 687 raft_consensus.cc:2241] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.635056 687 raft_consensus.cc:2270] T 72f57604705a4929815a3c7e3cd7756d P 67582185b6fe4ec9ba5423577fbb6c6b [term 2 FOLLOWER]: Raft consensus is shut down!
W20250626 08:22:35.658844 1455 consensus_peers.cc:487] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 -> Peer 642bad7878754413807793f3e5737d6a (127.0.171.193:43537): Couldn't send request to peer 642bad7878754413807793f3e5737d6a. Status: Network error: Client connection negotiation failed: client connection to 127.0.171.193:43537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:35.662628 687 tablet_server.cc:195] TabletServer@127.0.171.194:0 shutdown complete.
I20250626 08:22:35.695191 687 tablet_server.cc:178] TabletServer@127.0.171.195:0 shutting down...
I20250626 08:22:35.727449 687 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 08:22:35.728487 687 tablet_replica.cc:331] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.729691 687 raft_consensus.cc:2241] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.731338 687 raft_consensus.cc:2270] T aa78ce9017fd453d949ec53b92e6f065 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.733736 687 tablet_replica.cc:331] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.734375 687 raft_consensus.cc:2241] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.734937 687 raft_consensus.cc:2270] T c3774126f95c456dbe28c6d076e13910 P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.736742 687 tablet_replica.cc:331] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.737315 687 raft_consensus.cc:2241] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.737905 687 raft_consensus.cc:2270] T fd0ab0747ad14a24bed106da8471f77f P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.739681 687 tablet_replica.cc:331] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.740267 687 raft_consensus.cc:2241] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.740839 687 raft_consensus.cc:2270] T 72f57604705a4929815a3c7e3cd7756d P 5edae8a524254944a1b0d53a763a0f54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.742672 687 tablet_replica.cc:331] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.743116 687 raft_consensus.cc:2241] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.743645 687 raft_consensus.cc:2270] T 51d05dc235dc49e6927ccabc4dbe189e P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.745417 687 tablet_replica.cc:331] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.745851 687 raft_consensus.cc:2241] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 4 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.746979 687 raft_consensus.cc:2270] T 899475d7f1b345868d4ad1069d969127 P 5edae8a524254944a1b0d53a763a0f54 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.748790 687 tablet_replica.cc:331] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.749249 687 raft_consensus.cc:2241] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.749692 687 raft_consensus.cc:2270] T efd89843bdaa4ca293eca7af3af7e163 P 5edae8a524254944a1b0d53a763a0f54 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.751420 687 tablet_replica.cc:331] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.751827 687 raft_consensus.cc:2241] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.752235 687 raft_consensus.cc:2270] T 19171a3757464e19b2b1229496a4ab8d P 5edae8a524254944a1b0d53a763a0f54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.754057 687 tablet_replica.cc:331] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.754496 687 raft_consensus.cc:2241] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:35.754904 687 raft_consensus.cc:2270] T 25e1f15085a64706b7bfc464c12937a0 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.756521 687 tablet_replica.cc:331] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54: stopping tablet replica
I20250626 08:22:35.756907 687 raft_consensus.cc:2241] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 3 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.757876 687 raft_consensus.cc:2270] T 266c5c09162147c39240daff80116ee9 P 5edae8a524254944a1b0d53a763a0f54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.809185 687 tablet_server.cc:195] TabletServer@127.0.171.195:0 shutdown complete.
I20250626 08:22:35.830133 687 master.cc:561] Master@127.0.171.254:35743 shutting down...
I20250626 08:22:35.852991 687 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 1 LEADER]: Raft consensus shutting down.
I20250626 08:22:35.853626 687 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:35.853899 687 tablet_replica.cc:331] T 00000000000000000000000000000000 P 814ce75c1bb94487bfc9ef2b4b26ff22: stopping tablet replica
I20250626 08:22:35.873598 687 master.cc:583] Master@127.0.171.254:35743 shutdown complete.
5edae8a524254944a1b0d53a763a0f54 1
642bad7878754413807793f3e5737d6a 5
67582185b6fe4ec9ba5423577fbb6c6b 4
5edae8a524254944a1b0d53a763a0f54 3
642bad7878754413807793f3e5737d6a 3
67582185b6fe4ec9ba5423577fbb6c6b 4
5edae8a524254944a1b0d53a763a0f54 1
642bad7878754413807793f3e5737d6a 8
67582185b6fe4ec9ba5423577fbb6c6b 1
5edae8a524254944a1b0d53a763a0f54 3
642bad7878754413807793f3e5737d6a 4
67582185b6fe4ec9ba5423577fbb6c6b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29188 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250626 08:22:35.917811 687 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.171.254:42921
I20250626 08:22:35.918954 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:35.924309 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:35.925323 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:35.924526 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:35.927242 687 server_base.cc:1048] running on GCE node
I20250626 08:22:35.928225 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:35.928408 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:35.928521 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926155928510 us; error 0 us; skew 500 ppm
I20250626 08:22:35.928978 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:35.936093 687 webserver.cc:469] Webserver started at http://127.0.171.254:45343/ using document root <none> and password file <none>
I20250626 08:22:35.936554 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:35.936713 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:35.936955 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:35.938238 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/master-0-root/instance:
uuid: "12239b0901474fcc8d88fc20bc33dc1f"
format_stamp: "Formatted at 2025-06-26 08:22:35 on dist-test-slave-660p"
I20250626 08:22:35.942759 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20250626 08:22:35.945895 1729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:35.946643 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250626 08:22:35.946892 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/master-0-root
uuid: "12239b0901474fcc8d88fc20bc33dc1f"
format_stamp: "Formatted at 2025-06-26 08:22:35 on dist-test-slave-660p"
I20250626 08:22:35.947130 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:35.957648 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:35.958927 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:35.997201 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.254:42921
I20250626 08:22:35.997295 1781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.254:42921 every 8 connection(s)
I20250626 08:22:36.001010 1782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.011489 1782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Bootstrap starting.
I20250626 08:22:36.016148 1782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:36.020284 1782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: No bootstrap required, opened a new log
I20250626 08:22:36.022419 1782 raft_consensus.cc:357] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER }
I20250626 08:22:36.022841 1782 raft_consensus.cc:383] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:36.023034 1782 raft_consensus.cc:738] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12239b0901474fcc8d88fc20bc33dc1f, State: Initialized, Role: FOLLOWER
I20250626 08:22:36.023479 1782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER }
I20250626 08:22:36.023895 1782 raft_consensus.cc:397] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 08:22:36.024080 1782 raft_consensus.cc:491] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 08:22:36.024297 1782 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:36.028548 1782 raft_consensus.cc:513] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER }
I20250626 08:22:36.029017 1782 leader_election.cc:304] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12239b0901474fcc8d88fc20bc33dc1f; no voters:
I20250626 08:22:36.030162 1782 leader_election.cc:290] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 08:22:36.030531 1785 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:36.031839 1785 raft_consensus.cc:695] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [term 1 LEADER]: Becoming Leader. State: Replica: 12239b0901474fcc8d88fc20bc33dc1f, State: Running, Role: LEADER
I20250626 08:22:36.032545 1785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER }
I20250626 08:22:36.033154 1782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [sys.catalog]: configured and running, proceeding with master startup.
I20250626 08:22:36.038555 1786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12239b0901474fcc8d88fc20bc33dc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER } }
I20250626 08:22:36.038858 1787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12239b0901474fcc8d88fc20bc33dc1f. Latest consensus state: current_term: 1 leader_uuid: "12239b0901474fcc8d88fc20bc33dc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12239b0901474fcc8d88fc20bc33dc1f" member_type: VOTER } }
I20250626 08:22:36.039179 1786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:36.039317 1787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f [sys.catalog]: This master's current role is: LEADER
I20250626 08:22:36.041290 1792 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 08:22:36.049446 1792 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 08:22:36.050477 687 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 08:22:36.058580 1792 catalog_manager.cc:1349] Generated new cluster ID: 9621e03eac004624ad1a01ba182bdc6f
I20250626 08:22:36.058837 1792 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 08:22:36.083833 1792 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 08:22:36.085104 1792 catalog_manager.cc:1506] Loading token signing keys...
I20250626 08:22:36.097420 1792 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Generated new TSK 0
I20250626 08:22:36.097987 1792 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 08:22:36.116561 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:36.122661 1803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:36.123481 1804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:36.124974 1806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:36.125672 687 server_base.cc:1048] running on GCE node
I20250626 08:22:36.126740 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:36.126922 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:36.127105 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926156127085 us; error 0 us; skew 500 ppm
I20250626 08:22:36.127640 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:36.129962 687 webserver.cc:469] Webserver started at http://127.0.171.193:37671/ using document root <none> and password file <none>
I20250626 08:22:36.130437 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:36.130611 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:36.130857 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:36.131960 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/instance:
uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.136521 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250626 08:22:36.139716 1811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.140415 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250626 08:22:36.140954 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root
uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.141259 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:36.154574 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:36.155826 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:36.157274 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:36.159557 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:36.159775 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.160013 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:36.160178 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.205269 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.193:43311
I20250626 08:22:36.205364 1875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.193:43311 every 8 connection(s)
I20250626 08:22:36.209966 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:36.217890 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:36.218346 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:36.221441 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:36.221800 687 server_base.cc:1048] running on GCE node
I20250626 08:22:36.223858 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:36.224140 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:36.224380 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926156224359 us; error 0 us; skew 500 ppm
I20250626 08:22:36.225023 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:36.225659 1876 heartbeater.cc:344] Connected to a master server at 127.0.171.254:42921
I20250626 08:22:36.226059 1876 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:36.226907 1876 heartbeater.cc:507] Master 127.0.171.254:42921 requested a full tablet report, sending...
I20250626 08:22:36.227813 687 webserver.cc:469] Webserver started at http://127.0.171.194:33865/ using document root <none> and password file <none>
I20250626 08:22:36.228395 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:36.228638 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:36.228966 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:36.229141 1746 ts_manager.cc:194] Registered new tserver with Master: 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:36.230608 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/instance:
uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.231652 1746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32850
I20250626 08:22:36.235792 687 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250626 08:22:36.239096 1889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.239914 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250626 08:22:36.240185 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root
uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.240455 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:36.255981 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:36.257162 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:36.258612 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:36.260740 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:36.260941 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.261164 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:36.261343 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.299674 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.194:39813
I20250626 08:22:36.299791 1952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.194:39813 every 8 connection(s)
I20250626 08:22:36.304162 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:36.311085 1956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:36.312811 1957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:36.314370 687 server_base.cc:1048] running on GCE node
W20250626 08:22:36.315456 1960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:36.315867 1953 heartbeater.cc:344] Connected to a master server at 127.0.171.254:42921
I20250626 08:22:36.316206 1953 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:36.316430 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:36.316651 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:36.316802 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926156316783 us; error 0 us; skew 500 ppm
I20250626 08:22:36.316927 1953 heartbeater.cc:507] Master 127.0.171.254:42921 requested a full tablet report, sending...
I20250626 08:22:36.317349 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:36.319104 1746 ts_manager.cc:194] Registered new tserver with Master: 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:36.320039 687 webserver.cc:469] Webserver started at http://127.0.171.195:40279/ using document root <none> and password file <none>
I20250626 08:22:36.320518 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:36.320573 1746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32866
I20250626 08:22:36.320716 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:36.321066 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:36.322361 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/instance:
uuid: "755d83ca0913406b816e95415d502055"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.327288 687 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250626 08:22:36.330768 1964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.331686 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250626 08:22:36.331993 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root
uuid: "755d83ca0913406b816e95415d502055"
format_stamp: "Formatted at 2025-06-26 08:22:36 on dist-test-slave-660p"
I20250626 08:22:36.332286 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:36.344300 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:36.345507 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:36.346997 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:36.349339 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:36.349557 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:36.349789 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:36.349970 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250626 08:22:36.390928 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.195:38293
I20250626 08:22:36.391036 2027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.195:38293 every 8 connection(s)
I20250626 08:22:36.404058 2028 heartbeater.cc:344] Connected to a master server at 127.0.171.254:42921
I20250626 08:22:36.404422 2028 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:36.405113 2028 heartbeater.cc:507] Master 127.0.171.254:42921 requested a full tablet report, sending...
I20250626 08:22:36.406968 1746 ts_manager.cc:194] Registered new tserver with Master: 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:36.407176 687 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013218682s
I20250626 08:22:36.408242 1746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32876
I20250626 08:22:36.409519 687 test_util.cc:276] Using random seed: 1133816472
I20250626 08:22:36.432260 1746 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32880:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 08:22:36.438648 1746 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 08:22:36.669008 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 94ba4a836fec402199d1aecf04c04b62 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250626 08:22:36.670555 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ba4a836fec402199d1aecf04c04b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.671046 1915 tablet_service.cc:1468] Processing CreateTablet for tablet 5329c15e45ba48438209a93439414108 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250626 08:22:36.672437 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5329c15e45ba48438209a93439414108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.672521 1838 tablet_service.cc:1468] Processing CreateTablet for tablet 5329c15e45ba48438209a93439414108 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250626 08:22:36.673972 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5329c15e45ba48438209a93439414108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.673775 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 8e262c3543404a9cb840169098735ea2 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250626 08:22:36.674682 1834 tablet_service.cc:1468] Processing CreateTablet for tablet ac6c556e015045ac949f4ee3d2048b56 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250626 08:22:36.675096 1914 tablet_service.cc:1468] Processing CreateTablet for tablet 2717a332547d4ee8912bdda5311cffd8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250626 08:22:36.676380 1914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2717a332547d4ee8912bdda5311cffd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.676025 1835 tablet_service.cc:1468] Processing CreateTablet for tablet 5697952652a3463cbe2ee01e18348697 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250626 08:22:36.676218 1837 tablet_service.cc:1468] Processing CreateTablet for tablet 2717a332547d4ee8912bdda5311cffd8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250626 08:22:36.678094 1839 tablet_service.cc:1468] Processing CreateTablet for tablet 94ba4a836fec402199d1aecf04c04b62 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250626 08:22:36.678339 1832 tablet_service.cc:1468] Processing CreateTablet for tablet bcf570bd4fbe46dc9453a7319f45a376 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250626 08:22:36.678112 1833 tablet_service.cc:1468] Processing CreateTablet for tablet 171f89014c824d17a510a3cc17b3af0d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250626 08:22:36.679062 1913 tablet_service.cc:1468] Processing CreateTablet for tablet 8e262c3543404a9cb840169098735ea2 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250626 08:22:36.680397 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e262c3543404a9cb840169098735ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.680349 1840 tablet_service.cc:1468] Processing CreateTablet for tablet dc905d51bc4a450484091ffa39379396 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250626 08:22:36.682626 1831 tablet_service.cc:1468] Processing CreateTablet for tablet fec12ab89e754fea8f2b9e353bba3dee (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250626 08:22:36.682992 1912 tablet_service.cc:1468] Processing CreateTablet for tablet 5697952652a3463cbe2ee01e18348697 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250626 08:22:36.684242 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5697952652a3463cbe2ee01e18348697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.687207 1911 tablet_service.cc:1468] Processing CreateTablet for tablet ac6c556e015045ac949f4ee3d2048b56 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250626 08:22:36.688659 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c556e015045ac949f4ee3d2048b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.688365 1917 tablet_service.cc:1468] Processing CreateTablet for tablet dc905d51bc4a450484091ffa39379396 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250626 08:22:36.689612 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc905d51bc4a450484091ffa39379396. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.675103 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e262c3543404a9cb840169098735ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.690426 1833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171f89014c824d17a510a3cc17b3af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.692675 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5697952652a3463cbe2ee01e18348697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.694818 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 171f89014c824d17a510a3cc17b3af0d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250626 08:22:36.696184 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171f89014c824d17a510a3cc17b3af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.698138 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2717a332547d4ee8912bdda5311cffd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.698799 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c556e015045ac949f4ee3d2048b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.699414 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec12ab89e754fea8f2b9e353bba3dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.702217 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc905d51bc4a450484091ffa39379396. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.703692 1909 tablet_service.cc:1468] Processing CreateTablet for tablet bcf570bd4fbe46dc9453a7319f45a376 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250626 08:22:36.703994 1908 tablet_service.cc:1468] Processing CreateTablet for tablet fec12ab89e754fea8f2b9e353bba3dee (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250626 08:22:36.705124 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf570bd4fbe46dc9453a7319f45a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.705667 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ba4a836fec402199d1aecf04c04b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.707865 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf570bd4fbe46dc9453a7319f45a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.710597 1908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec12ab89e754fea8f2b9e353bba3dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.727191 1990 tablet_service.cc:1468] Processing CreateTablet for tablet 5329c15e45ba48438209a93439414108 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250626 08:22:36.727524 1986 tablet_service.cc:1468] Processing CreateTablet for tablet ac6c556e015045ac949f4ee3d2048b56 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250626 08:22:36.728688 1990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5329c15e45ba48438209a93439414108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.729889 1988 tablet_service.cc:1468] Processing CreateTablet for tablet 8e262c3543404a9cb840169098735ea2 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250626 08:22:36.731268 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e262c3543404a9cb840169098735ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.731019 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 94ba4a836fec402199d1aecf04c04b62 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250626 08:22:36.732322 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ba4a836fec402199d1aecf04c04b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.733232 1989 tablet_service.cc:1468] Processing CreateTablet for tablet 2717a332547d4ee8912bdda5311cffd8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250626 08:22:36.735275 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c556e015045ac949f4ee3d2048b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.727191 1992 tablet_service.cc:1468] Processing CreateTablet for tablet dc905d51bc4a450484091ffa39379396 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250626 08:22:36.736302 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 5697952652a3463cbe2ee01e18348697 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250626 08:22:36.736281 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc905d51bc4a450484091ffa39379396. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.740249 1989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2717a332547d4ee8912bdda5311cffd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.741760 1984 tablet_service.cc:1468] Processing CreateTablet for tablet bcf570bd4fbe46dc9453a7319f45a376 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250626 08:22:36.733880 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 3d1eb5f502d8430b9109e246276cc44e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250626 08:22:36.744967 1985 tablet_service.cc:1468] Processing CreateTablet for tablet 171f89014c824d17a510a3cc17b3af0d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250626 08:22:36.745306 1983 tablet_service.cc:1468] Processing CreateTablet for tablet fec12ab89e754fea8f2b9e353bba3dee (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250626 08:22:36.748396 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1eb5f502d8430b9109e246276cc44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.762285 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf570bd4fbe46dc9453a7319f45a376. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.766166 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec12ab89e754fea8f2b9e353bba3dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.770262 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5697952652a3463cbe2ee01e18348697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.774241 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171f89014c824d17a510a3cc17b3af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.798292 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 3d1eb5f502d8430b9109e246276cc44e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250626 08:22:36.799652 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1eb5f502d8430b9109e246276cc44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.801687 1913 tablet_service.cc:1468] Processing CreateTablet for tablet 3e928895bcfa4d00823f3f4b50eebf86 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250626 08:22:36.802968 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e928895bcfa4d00823f3f4b50eebf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.829869 1831 tablet_service.cc:1468] Processing CreateTablet for tablet 3e928895bcfa4d00823f3f4b50eebf86 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250626 08:22:36.831211 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e928895bcfa4d00823f3f4b50eebf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.855222 2043 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:36.862870 1835 tablet_service.cc:1468] Processing CreateTablet for tablet cca900160913464799d0cfa9d2866779 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250626 08:22:36.864326 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca900160913464799d0cfa9d2866779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.870760 1914 tablet_service.cc:1468] Processing CreateTablet for tablet cca900160913464799d0cfa9d2866779 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250626 08:22:36.872056 1914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca900160913464799d0cfa9d2866779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.877939 1909 tablet_service.cc:1468] Processing CreateTablet for tablet b498813b859f40cd95ed6adf6a2d8918 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250626 08:22:36.879241 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b498813b859f40cd95ed6adf6a2d8918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.883952 1908 tablet_service.cc:1468] Processing CreateTablet for tablet 331d5bdb3fda41489445725e41b94bd1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250626 08:22:36.885207 1912 tablet_service.cc:1468] Processing CreateTablet for tablet cef3ff80065343cdbaf58ef8b1bbedf9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250626 08:22:36.885743 1908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331d5bdb3fda41489445725e41b94bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.888059 1834 tablet_service.cc:1468] Processing CreateTablet for tablet cef3ff80065343cdbaf58ef8b1bbedf9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250626 08:22:36.889252 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef3ff80065343cdbaf58ef8b1bbedf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.872069 1838 tablet_service.cc:1468] Processing CreateTablet for tablet b498813b859f40cd95ed6adf6a2d8918 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250626 08:22:36.895195 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b498813b859f40cd95ed6adf6a2d8918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.897609 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef3ff80065343cdbaf58ef8b1bbedf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.899039 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 2e6a8a739b954cc082a290e6c529a271 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250626 08:22:36.900240 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6a8a739b954cc082a290e6c529a271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.900830 1917 tablet_service.cc:1468] Processing CreateTablet for tablet cd7130b92c4840deaca578e523eb4a09 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250626 08:22:36.911167 1837 tablet_service.cc:1468] Processing CreateTablet for tablet 331d5bdb3fda41489445725e41b94bd1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250626 08:22:36.911550 2044 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:36.922473 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 3d1eb5f502d8430b9109e246276cc44e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250626 08:22:36.919680 1839 tablet_service.cc:1468] Processing CreateTablet for tablet 2e6a8a739b954cc082a290e6c529a271 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250626 08:22:36.929435 2044 tablet_bootstrap.cc:654] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:36.925110 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6a8a739b954cc082a290e6c529a271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.908762 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7130b92c4840deaca578e523eb4a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.940124 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331d5bdb3fda41489445725e41b94bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.942152 1983 tablet_service.cc:1468] Processing CreateTablet for tablet 3e928895bcfa4d00823f3f4b50eebf86 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250626 08:22:36.957278 2043 tablet_bootstrap.cc:654] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:36.957747 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1eb5f502d8430b9109e246276cc44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.964322 1840 tablet_service.cc:1468] Processing CreateTablet for tablet cd7130b92c4840deaca578e523eb4a09 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250626 08:22:36.966987 1911 tablet_service.cc:1468] Processing CreateTablet for tablet 75222ddacb444f7ba27de92b1571ada8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250626 08:22:36.970636 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75222ddacb444f7ba27de92b1571ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.972932 1992 tablet_service.cc:1468] Processing CreateTablet for tablet 331d5bdb3fda41489445725e41b94bd1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250626 08:22:36.967989 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e928895bcfa4d00823f3f4b50eebf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.974216 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331d5bdb3fda41489445725e41b94bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.975566 1915 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9d96212a6463eb1aa71d0b3d300a6 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250626 08:22:36.976840 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9d96212a6463eb1aa71d0b3d300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.980118 1832 tablet_service.cc:1468] Processing CreateTablet for tablet 75222ddacb444f7ba27de92b1571ada8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250626 08:22:36.965667 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7130b92c4840deaca578e523eb4a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:36.969712 1988 tablet_service.cc:1468] Processing CreateTablet for tablet cca900160913464799d0cfa9d2866779 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250626 08:22:36.999770 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca900160913464799d0cfa9d2866779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.016314 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75222ddacb444f7ba27de92b1571ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.018975 1984 tablet_service.cc:1468] Processing CreateTablet for tablet 2e6a8a739b954cc082a290e6c529a271 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250626 08:22:37.020320 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6a8a739b954cc082a290e6c529a271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.020448 1987 tablet_service.cc:1468] Processing CreateTablet for tablet cef3ff80065343cdbaf58ef8b1bbedf9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250626 08:22:37.021871 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef3ff80065343cdbaf58ef8b1bbedf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.026239 1985 tablet_service.cc:1468] Processing CreateTablet for tablet b498813b859f40cd95ed6adf6a2d8918 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250626 08:22:37.027688 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b498813b859f40cd95ed6adf6a2d8918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.037889 1908 tablet_service.cc:1468] Processing CreateTablet for tablet a71e0323b96e4a47b1ba2e0a3d72840f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250626 08:22:37.038389 2044 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:37.039746 2044 ts_tablet_manager.cc:1397] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.129s user 0.017s sys 0.017s
I20250626 08:22:37.040390 1986 tablet_service.cc:1468] Processing CreateTablet for tablet 75222ddacb444f7ba27de92b1571ada8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250626 08:22:37.040393 1990 tablet_service.cc:1468] Processing CreateTablet for tablet cd7130b92c4840deaca578e523eb4a09 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250626 08:22:37.041785 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75222ddacb444f7ba27de92b1571ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.042749 2044 raft_consensus.cc:357] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:37.045296 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:37.037541 1909 tablet_service.cc:1468] Processing CreateTablet for tablet 0899df8ec53c45498dd1f3da415d5155 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250626 08:22:37.046273 1990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7130b92c4840deaca578e523eb4a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.046836 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0899df8ec53c45498dd1f3da415d5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.048997 2045 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:37.050709 1833 tablet_service.cc:1468] Processing CreateTablet for tablet a71e0323b96e4a47b1ba2e0a3d72840f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250626 08:22:37.050895 1839 tablet_service.cc:1468] Processing CreateTablet for tablet 0899df8ec53c45498dd1f3da415d5155 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250626 08:22:37.052066 1833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71e0323b96e4a47b1ba2e0a3d72840f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.056108 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 0914c29163894ada9540f1133a461648 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250626 08:22:37.057479 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0914c29163894ada9540f1133a461648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.059377 1832 tablet_service.cc:1468] Processing CreateTablet for tablet 148557a00636474cad8673c1f02aae5e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250626 08:22:37.060271 1913 tablet_service.cc:1468] Processing CreateTablet for tablet 148557a00636474cad8673c1f02aae5e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250626 08:22:37.060730 1908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71e0323b96e4a47b1ba2e0a3d72840f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.061550 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148557a00636474cad8673c1f02aae5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.043613 2044 raft_consensus.cc:383] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.063891 2044 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.064616 1831 tablet_service.cc:1468] Processing CreateTablet for tablet 187f08b11130495ba8e2a482370447f4 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250626 08:22:37.064627 2044 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.067180 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 187f08b11130495ba8e2a482370447f4 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250626 08:22:37.068727 1837 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9d96212a6463eb1aa71d0b3d300a6 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250626 08:22:37.070289 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 4be9d96212a6463eb1aa71d0b3d300a6 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250626 08:22:37.071584 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9d96212a6463eb1aa71d0b3d300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.071118 1914 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31173b902a46949ba221ed157ca482 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250626 08:22:37.064075 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 0914c29163894ada9540f1133a461648 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250626 08:22:37.073848 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0914c29163894ada9540f1133a461648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.074700 1835 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31173b902a46949ba221ed157ca482 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250626 08:22:37.075846 2045 tablet_bootstrap.cc:654] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.075963 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31173b902a46949ba221ed157ca482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.078419 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148557a00636474cad8673c1f02aae5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.079406 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0899df8ec53c45498dd1f3da415d5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.081647 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187f08b11130495ba8e2a482370447f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.083107 1834 tablet_service.cc:1468] Processing CreateTablet for tablet c014cb521ac2424fa52882a7d676dbec (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250626 08:22:37.084319 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c014cb521ac2424fa52882a7d676dbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.098037 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9d96212a6463eb1aa71d0b3d300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.098881 1914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31173b902a46949ba221ed157ca482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.105469 1983 tablet_service.cc:1468] Processing CreateTablet for tablet 5f31173b902a46949ba221ed157ca482 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250626 08:22:37.108886 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187f08b11130495ba8e2a482370447f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:22:37.120589 1732 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 51098.9us
I20250626 08:22:37.133064 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 0899df8ec53c45498dd1f3da415d5155 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250626 08:22:37.117539 1988 tablet_service.cc:1468] Processing CreateTablet for tablet 187f08b11130495ba8e2a482370447f4 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250626 08:22:37.117595 1989 tablet_service.cc:1468] Processing CreateTablet for tablet 0914c29163894ada9540f1133a461648 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250626 08:22:37.145030 2043 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:37.145651 2043 ts_tablet_manager.cc:1397] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.291s user 0.006s sys 0.012s
I20250626 08:22:37.148433 2043 raft_consensus.cc:357] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.158092 2043 raft_consensus.cc:383] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.158524 2043 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.159453 2043 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.154734 1840 tablet_service.cc:1468] Processing CreateTablet for tablet 84dfea36e4274e44b9a893967d69fd7e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250626 08:22:37.161434 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84dfea36e4274e44b9a893967d69fd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.162016 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f31173b902a46949ba221ed157ca482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.185204 2043 ts_tablet_manager.cc:1428] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.039s user 0.005s sys 0.006s
I20250626 08:22:37.188411 1876 heartbeater.cc:499] Master 127.0.171.254:42921 was elected leader, sending a full tablet report...
I20250626 08:22:37.188541 2043 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:37.188455 1917 tablet_service.cc:1468] Processing CreateTablet for tablet 84dfea36e4274e44b9a893967d69fd7e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250626 08:22:37.189401 1992 tablet_service.cc:1468] Processing CreateTablet for tablet a71e0323b96e4a47b1ba2e0a3d72840f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250626 08:22:37.189841 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84dfea36e4274e44b9a893967d69fd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.192476 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 148557a00636474cad8673c1f02aae5e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250626 08:22:37.194259 1953 heartbeater.cc:499] Master 127.0.171.254:42921 was elected leader, sending a full tablet report...
I20250626 08:22:37.194648 2044 ts_tablet_manager.cc:1428] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.155s user 0.011s sys 0.004s
I20250626 08:22:37.194717 2043 tablet_bootstrap.cc:654] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.195813 2044 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:37.198817 1915 tablet_service.cc:1468] Processing CreateTablet for tablet c014cb521ac2424fa52882a7d676dbec (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250626 08:22:37.199507 1985 tablet_service.cc:1468] Processing CreateTablet for tablet c014cb521ac2424fa52882a7d676dbec (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250626 08:22:37.202641 1838 tablet_service.cc:1468] Processing CreateTablet for tablet f2bf9541d0634f9fab914afd74bb9505 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250626 08:22:37.203984 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bf9541d0634f9fab914afd74bb9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.216066 1911 tablet_service.cc:1468] Processing CreateTablet for tablet f2bf9541d0634f9fab914afd74bb9505 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250626 08:22:37.232115 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0899df8ec53c45498dd1f3da415d5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.249583 1917 tablet_service.cc:1468] Processing CreateTablet for tablet 70ce63b4769e4ca8980ddb20229389ef (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250626 08:22:37.250345 1912 tablet_service.cc:1468] Processing CreateTablet for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250626 08:22:37.251659 2048 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:37.253130 2048 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.257616 1835 tablet_service.cc:1468] Processing CreateTablet for tablet 423807ecf2d54e869ffc86a272ebd10a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250626 08:22:37.258550 2048 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:37.258985 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423807ecf2d54e869ffc86a272ebd10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.266098 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c014cb521ac2424fa52882a7d676dbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.252164 1989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0914c29163894ada9540f1133a461648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.273622 2045 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:37.274224 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148557a00636474cad8673c1f02aae5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.274441 2045 ts_tablet_manager.cc:1397] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.226s user 0.022s sys 0.038s
I20250626 08:22:37.270912 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bf9541d0634f9fab914afd74bb9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.281368 1832 tablet_service.cc:1468] Processing CreateTablet for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250626 08:22:37.282204 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 187f08b11130495ba8e2a482370447f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.282301 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.282858 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:22:37.296422 1732 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45245.2us
I20250626 08:22:37.298768 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 423807ecf2d54e869ffc86a272ebd10a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250626 08:22:37.294338 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c014cb521ac2424fa52882a7d676dbec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.302960 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71e0323b96e4a47b1ba2e0a3d72840f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.310791 1833 tablet_service.cc:1468] Processing CreateTablet for tablet 70ce63b4769e4ca8980ddb20229389ef (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250626 08:22:37.312651 1833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ce63b4769e4ca8980ddb20229389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.294735 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ce63b4769e4ca8980ddb20229389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.315128 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423807ecf2d54e869ffc86a272ebd10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.318462 2043 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:37.318971 2043 ts_tablet_manager.cc:1397] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.131s user 0.014s sys 0.013s
I20250626 08:22:37.321866 2043 raft_consensus.cc:357] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.322619 2043 raft_consensus.cc:383] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.322921 2043 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.288479 2045 raft_consensus.cc:357] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.334141 2045 raft_consensus.cc:383] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.334510 2045 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.334929 1909 tablet_service.cc:1468] Processing CreateTablet for tablet 46151fddbbf14be68feb086aa1fee4d9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250626 08:22:37.335294 2045 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.336323 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46151fddbbf14be68feb086aa1fee4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.363050 2044 tablet_bootstrap.cc:654] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.364475 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 46151fddbbf14be68feb086aa1fee4d9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250626 08:22:37.367012 1984 tablet_service.cc:1468] Processing CreateTablet for tablet 84dfea36e4274e44b9a893967d69fd7e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250626 08:22:37.370822 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84dfea36e4274e44b9a893967d69fd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.380029 1990 tablet_service.cc:1468] Processing CreateTablet for tablet 46151fddbbf14be68feb086aa1fee4d9 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250626 08:22:37.381319 1990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46151fddbbf14be68feb086aa1fee4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.374155 1992 tablet_service.cc:1468] Processing CreateTablet for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250626 08:22:37.374385 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 423807ecf2d54e869ffc86a272ebd10a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250626 08:22:37.375512 1838 tablet_service.cc:1468] Processing CreateTablet for tablet 9996474aab3b4a5ca0e10b9f200b2abc (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250626 08:22:37.376889 1915 tablet_service.cc:1468] Processing CreateTablet for tablet 9996474aab3b4a5ca0e10b9f200b2abc (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250626 08:22:37.385161 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423807ecf2d54e869ffc86a272ebd10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.389214 1911 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cb31b10924fbcb39dfbbcac77397e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250626 08:22:37.387210 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9996474aab3b4a5ca0e10b9f200b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.393951 1831 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cb31b10924fbcb39dfbbcac77397e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250626 08:22:37.381847 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 02e74341f4bd4b68b89e1472c394f5bb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250626 08:22:37.395107 2043 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.397693 2043 ts_tablet_manager.cc:1428] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.078s user 0.005s sys 0.001s
I20250626 08:22:37.400303 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46151fddbbf14be68feb086aa1fee4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.410773 2028 heartbeater.cc:499] Master 127.0.171.254:42921 was elected leader, sending a full tablet report...
I20250626 08:22:37.411473 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cb31b10924fbcb39dfbbcac77397e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.414214 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9996474aab3b4a5ca0e10b9f200b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.416899 1986 tablet_service.cc:1468] Processing CreateTablet for tablet f2bf9541d0634f9fab914afd74bb9505 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250626 08:22:37.365536 1834 tablet_service.cc:1468] Processing CreateTablet for tablet 02e74341f4bd4b68b89e1472c394f5bb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250626 08:22:37.395648 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e74341f4bd4b68b89e1472c394f5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.418956 1839 tablet_service.cc:1468] Processing CreateTablet for tablet c8539685c775453b8aaf4921fa90fa63 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250626 08:22:37.420228 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8539685c775453b8aaf4921fa90fa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.422580 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e74341f4bd4b68b89e1472c394f5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.391023 1989 tablet_service.cc:1468] Processing CreateTablet for tablet 70ce63b4769e4ca8980ddb20229389ef (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250626 08:22:37.428925 1837 tablet_service.cc:1468] Processing CreateTablet for tablet edeafceae4424506a7bffa7a9fbac201 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250626 08:22:37.430279 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edeafceae4424506a7bffa7a9fbac201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.435554 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:37.437357 2043 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
W20250626 08:22:37.438503 1814 leader_election.cc:343] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.439713 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.442595 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bf9541d0634f9fab914afd74bb9505. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.444453 1908 tablet_service.cc:1468] Processing CreateTablet for tablet edeafceae4424506a7bffa7a9fbac201 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250626 08:22:37.445704 1908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edeafceae4424506a7bffa7a9fbac201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.449494 1989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ce63b4769e4ca8980ddb20229389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.451773 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 9996474aab3b4a5ca0e10b9f200b2abc (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250626 08:22:37.451979 1913 tablet_service.cc:1468] Processing CreateTablet for tablet c8539685c775453b8aaf4921fa90fa63 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250626 08:22:37.453051 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9996474aab3b4a5ca0e10b9f200b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.453258 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8539685c775453b8aaf4921fa90fa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.454998 1840 tablet_service.cc:1468] Processing CreateTablet for tablet 856b2d0e8d2948bd9bfc33345dcd4886 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250626 08:22:37.456236 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b2d0e8d2948bd9bfc33345dcd4886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.460172 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:37.463253 1816 leader_election.cc:343] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.465860 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cb31b10924fbcb39dfbbcac77397e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.465512 1816 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:37.467262 1835 tablet_service.cc:1468] Processing CreateTablet for tablet 4f29b6256843489397f6e58f34751764 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250626 08:22:37.467782 2048 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:37.494758 1984 tablet_service.cc:1468] Processing CreateTablet for tablet 02e74341f4bd4b68b89e1472c394f5bb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250626 08:22:37.496313 1909 tablet_service.cc:1468] Processing CreateTablet for tablet 856b2d0e8d2948bd9bfc33345dcd4886 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250626 08:22:37.497535 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b2d0e8d2948bd9bfc33345dcd4886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.498215 1912 tablet_service.cc:1468] Processing CreateTablet for tablet 4f29b6256843489397f6e58f34751764 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250626 08:22:37.499475 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f29b6256843489397f6e58f34751764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.502444 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f29b6256843489397f6e58f34751764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.508630 1914 tablet_service.cc:1468] Processing CreateTablet for tablet 766debd72f3f4feaa9e81cd3690afdc1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250626 08:22:37.498294 2044 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:37.510128 1832 tablet_service.cc:1468] Processing CreateTablet for tablet 766debd72f3f4feaa9e81cd3690afdc1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250626 08:22:37.511341 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766debd72f3f4feaa9e81cd3690afdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.515522 1988 tablet_service.cc:1468] Processing CreateTablet for tablet edeafceae4424506a7bffa7a9fbac201 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250626 08:22:37.516258 2050 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:37.516729 2050 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.517113 1915 tablet_service.cc:1468] Processing CreateTablet for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250626 08:22:37.519526 2050 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:37.510282 2044 ts_tablet_manager.cc:1397] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.315s user 0.016s sys 0.004s
I20250626 08:22:37.532045 1985 tablet_service.cc:1468] Processing CreateTablet for tablet c8539685c775453b8aaf4921fa90fa63 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250626 08:22:37.534109 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 6f8cb31b10924fbcb39dfbbcac77397e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250626 08:22:37.535730 2045 ts_tablet_manager.cc:1428] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.261s user 0.004s sys 0.013s
I20250626 08:22:37.536777 2045 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:37.537010 2044 raft_consensus.cc:357] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.537830 2044 raft_consensus.cc:383] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.538273 2044 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.526015 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e74341f4bd4b68b89e1472c394f5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.543568 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8539685c775453b8aaf4921fa90fa63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.543524 2044 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.546155 2044 ts_tablet_manager.cc:1428] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250626 08:22:37.547518 2044 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:37.550252 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cb31b10924fbcb39dfbbcac77397e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.550824 2043 tablet_bootstrap.cc:654] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.553507 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 4c17e5edca1f4c3784180a1653f3234d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250626 08:22:37.558951 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edeafceae4424506a7bffa7a9fbac201. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.563097 1983 tablet_service.cc:1468] Processing CreateTablet for tablet 856b2d0e8d2948bd9bfc33345dcd4886 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250626 08:22:37.564455 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b2d0e8d2948bd9bfc33345dcd4886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.579088 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17e5edca1f4c3784180a1653f3234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.582247 1914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766debd72f3f4feaa9e81cd3690afdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.584841 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b805acc8ae4c3abfe6d6ee40fbf456. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.632742 1986 tablet_service.cc:1468] Processing CreateTablet for tablet 4f29b6256843489397f6e58f34751764 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250626 08:22:37.641005 2044 tablet_bootstrap.cc:654] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.653136 1913 tablet_service.cc:1468] Processing CreateTablet for tablet 178e69a3d3834cae8db6684d32c1c4ce (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250626 08:22:37.655134 2045 tablet_bootstrap.cc:654] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.666301 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f29b6256843489397f6e58f34751764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.670544 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178e69a3d3834cae8db6684d32c1c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.675696 1838 tablet_service.cc:1468] Processing CreateTablet for tablet 4c17e5edca1f4c3784180a1653f3234d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250626 08:22:37.676980 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17e5edca1f4c3784180a1653f3234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.677222 1835 tablet_service.cc:1468] Processing CreateTablet for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250626 08:22:37.680971 1908 tablet_service.cc:1468] Processing CreateTablet for tablet 3946d60648894026984085c3abee51ea (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250626 08:22:37.682880 1917 tablet_service.cc:1468] Processing CreateTablet for tablet 28139bf29a374c98a8fec4eddf814fac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250626 08:22:37.684073 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28139bf29a374c98a8fec4eddf814fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.686648 1908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3946d60648894026984085c3abee51ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.703562 2043 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:37.704128 2043 ts_tablet_manager.cc:1397] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.267s user 0.005s sys 0.022s
I20250626 08:22:37.706437 1834 tablet_service.cc:1468] Processing CreateTablet for tablet 178e69a3d3834cae8db6684d32c1c4ce (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250626 08:22:37.706923 2043 raft_consensus.cc:357] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.707682 2043 raft_consensus.cc:383] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.707989 2043 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.710819 1837 tablet_service.cc:1468] Processing CreateTablet for tablet 75b98c18e48c4076ba1d545113f76522 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250626 08:22:37.716486 1992 tablet_service.cc:1468] Processing CreateTablet for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250626 08:22:37.717269 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 8353d62d68914958b13bb87541c357f8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250626 08:22:37.717799 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b805acc8ae4c3abfe6d6ee40fbf456. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.720927 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b805acc8ae4c3abfe6d6ee40fbf456. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.721676 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b98c18e48c4076ba1d545113f76522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.722752 1833 tablet_service.cc:1468] Processing CreateTablet for tablet 28139bf29a374c98a8fec4eddf814fac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250626 08:22:37.724038 1833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28139bf29a374c98a8fec4eddf814fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.726265 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8353d62d68914958b13bb87541c357f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.711066 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 3946d60648894026984085c3abee51ea (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250626 08:22:37.727166 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3946d60648894026984085c3abee51ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.733384 1840 tablet_service.cc:1468] Processing CreateTablet for tablet 8353d62d68914958b13bb87541c357f8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250626 08:22:37.734781 1912 tablet_service.cc:1468] Processing CreateTablet for tablet 75b98c18e48c4076ba1d545113f76522 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250626 08:22:37.735846 2050 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:37.736131 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b98c18e48c4076ba1d545113f76522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.736310 2050 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.738446 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8353d62d68914958b13bb87541c357f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.738250 2050 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:37.740408 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:37.742316 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:37.744303 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:37.738229 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 766debd72f3f4feaa9e81cd3690afdc1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250626 08:22:37.746656 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 766debd72f3f4feaa9e81cd3690afdc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.748587 1831 tablet_service.cc:1468] Processing CreateTablet for tablet 26cfef3699a141f8b7686ae2b75a3beb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250626 08:22:37.754854 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:37.755548 1909 tablet_service.cc:1468] Processing CreateTablet for tablet 7846e33f4b874a229eb7250f99a6db61 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250626 08:22:37.756783 1909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7846e33f4b874a229eb7250f99a6db61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.757513 1989 tablet_service.cc:1468] Processing CreateTablet for tablet 4c17e5edca1f4c3784180a1653f3234d (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250626 08:22:37.758828 1989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17e5edca1f4c3784180a1653f3234d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.708729 1839 tablet_service.cc:1468] Processing CreateTablet for tablet 7846e33f4b874a229eb7250f99a6db61 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20250626 08:22:37.759599 1893 leader_election.cc:343] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.771538 1911 tablet_service.cc:1468] Processing CreateTablet for tablet 26cfef3699a141f8b7686ae2b75a3beb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250626 08:22:37.773712 2045 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
W20250626 08:22:37.760325 1893 leader_election.cc:343] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.776319 2045 ts_tablet_manager.cc:1397] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.240s user 0.021s sys 0.016s
I20250626 08:22:37.777136 2044 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:37.795073 2044 ts_tablet_manager.cc:1397] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.248s user 0.020s sys 0.003s
I20250626 08:22:37.797307 2045 raft_consensus.cc:357] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.778118 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cfef3699a141f8b7686ae2b75a3beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.798120 2045 raft_consensus.cc:383] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.798440 2045 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.799252 2045 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.801695 2045 ts_tablet_manager.cc:1428] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250626 08:22:37.774497 2043 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.793646 2048 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:37.774523 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178e69a3d3834cae8db6684d32c1c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.810019 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7846e33f4b874a229eb7250f99a6db61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.811267 1985 tablet_service.cc:1468] Processing CreateTablet for tablet 75b98c18e48c4076ba1d545113f76522 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250626 08:22:37.812479 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b98c18e48c4076ba1d545113f76522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.812945 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cfef3699a141f8b7686ae2b75a3beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.797667 2044 raft_consensus.cc:357] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.814405 2044 raft_consensus.cc:383] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:37.814733 2044 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:37.815421 2044 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.816205 1917 tablet_service.cc:1468] Processing CreateTablet for tablet d9f49ece70144b92b201204afc3042ed (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250626 08:22:37.792253 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 7846e33f4b874a229eb7250f99a6db61 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250626 08:22:37.808507 2048 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.817461 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f49ece70144b92b201204afc3042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.818372 2048 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:37.822265 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:37.823760 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:37.824918 1986 tablet_service.cc:1468] Processing CreateTablet for tablet 3946d60648894026984085c3abee51ea (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250626 08:22:37.825181 1837 tablet_service.cc:1468] Processing CreateTablet for tablet 736124bc239f4257b44dfc38e3acc17f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20250626 08:22:37.825361 1814 leader_election.cc:343] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.826190 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3946d60648894026984085c3abee51ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.826457 2045 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
W20250626 08:22:37.827064 1816 leader_election.cc:343] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.827517 1816 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:37.829298 2048 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:37.829843 2056 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:37.830309 2056 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:37.832213 1983 tablet_service.cc:1468] Processing CreateTablet for tablet 28139bf29a374c98a8fec4eddf814fac (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250626 08:22:37.832710 2056 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:37.801599 1990 tablet_service.cc:1468] Processing CreateTablet for tablet 178e69a3d3834cae8db6684d32c1c4ce (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250626 08:22:37.834986 2043 ts_tablet_manager.cc:1428] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.131s user 0.009s sys 0.000s
I20250626 08:22:37.829391 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7846e33f4b874a229eb7250f99a6db61. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:22:37.807685 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 47418.1us
I20250626 08:22:37.866276 1837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736124bc239f4257b44dfc38e3acc17f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:22:37.861670 1893 leader_election.cc:343] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.870203 1893 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:37.870612 1832 tablet_service.cc:1468] Processing CreateTablet for tablet d9f49ece70144b92b201204afc3042ed (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20250626 08:22:37.871392 1893 leader_election.cc:343] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:37.871934 1832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f49ece70144b92b201204afc3042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.871841 1893 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:37.857077 1913 tablet_service.cc:1468] Processing CreateTablet for tablet 736124bc239f4257b44dfc38e3acc17f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250626 08:22:37.871023 2050 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:37.860256 1988 tablet_service.cc:1468] Processing CreateTablet for tablet 26cfef3699a141f8b7686ae2b75a3beb (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250626 08:22:37.874907 2050 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:37.877096 1913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736124bc239f4257b44dfc38e3acc17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.881454 2045 tablet_bootstrap.cc:654] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.881896 1910 tablet_service.cc:1468] Processing CreateTablet for tablet 66ab2bec395d46f2a6a77385c304da93 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250626 08:22:37.883144 1910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ab2bec395d46f2a6a77385c304da93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.884928 2043 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:37.892657 1838 tablet_service.cc:1468] Processing CreateTablet for tablet 66ab2bec395d46f2a6a77385c304da93 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250626 08:22:37.894680 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28139bf29a374c98a8fec4eddf814fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.900108 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cfef3699a141f8b7686ae2b75a3beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.903851 2044 ts_tablet_manager.cc:1428] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.108s user 0.007s sys 0.005s
I20250626 08:22:37.907053 1839 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7d84adc07945fe8a3aa16f71b111e3 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250626 08:22:37.910275 1914 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7d84adc07945fe8a3aa16f71b111e3 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250626 08:22:37.911623 1914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7d84adc07945fe8a3aa16f71b111e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.913597 1916 tablet_service.cc:1468] Processing CreateTablet for tablet 4a65a55472e64be4b9ae56426278c3c1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250626 08:22:37.914948 1916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a65a55472e64be4b9ae56426278c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.915587 1911 tablet_service.cc:1468] Processing CreateTablet for tablet 9122458a1eae4f07b1a7f4ff63d6be19 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250626 08:22:37.916692 2043 tablet_bootstrap.cc:654] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:37.916808 1911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9122458a1eae4f07b1a7f4ff63d6be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.912343 1915 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2fc654780843abbda82ba8f5df515e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250626 08:22:37.918898 1984 tablet_service.cc:1468] Processing CreateTablet for tablet 8353d62d68914958b13bb87541c357f8 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250626 08:22:37.896214 1838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ab2bec395d46f2a6a77385c304da93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.925513 1990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 178e69a3d3834cae8db6684d32c1c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.925721 2044 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:37.935269 1831 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2fc654780843abbda82ba8f5df515e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250626 08:22:37.933250 1915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2fc654780843abbda82ba8f5df515e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.944022 1840 tablet_service.cc:1468] Processing CreateTablet for tablet 4a65a55472e64be4b9ae56426278c3c1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250626 08:22:37.948871 1839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7d84adc07945fe8a3aa16f71b111e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.953606 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8353d62d68914958b13bb87541c357f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.971906 1917 tablet_service.cc:1468] Processing CreateTablet for tablet 7a16389239e54c52bb619a62b1b0b91a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
W20250626 08:22:37.948117 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 76756.4us
I20250626 08:22:37.975443 1917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a16389239e54c52bb619a62b1b0b91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.975175 1836 tablet_service.cc:1468] Processing CreateTablet for tablet 7a16389239e54c52bb619a62b1b0b91a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250626 08:22:37.981642 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:37.983294 1912 tablet_service.cc:1468] Processing CreateTablet for tablet 5549f2d4a32b4170ad59fc7489d1365f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250626 08:22:38.027616 2056 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.028041 1912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5549f2d4a32b4170ad59fc7489d1365f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.028229 2056 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.030301 2056 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.031683 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:37.994582 1835 tablet_service.cc:1468] Processing CreateTablet for tablet 9122458a1eae4f07b1a7f4ff63d6be19 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250626 08:22:38.006228 1831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2fc654780843abbda82ba8f5df515e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.008737 1834 tablet_service.cc:1468] Processing CreateTablet for tablet 5549f2d4a32b4170ad59fc7489d1365f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250626 08:22:38.013774 1989 tablet_service.cc:1468] Processing CreateTablet for tablet d9f49ece70144b92b201204afc3042ed (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250626 08:22:38.034067 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.034332 1834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5549f2d4a32b4170ad59fc7489d1365f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.035073 1989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9f49ece70144b92b201204afc3042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:37.995062 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 736124bc239f4257b44dfc38e3acc17f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250626 08:22:38.020779 1986 tablet_service.cc:1468] Processing CreateTablet for tablet 7e7d84adc07945fe8a3aa16f71b111e3 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250626 08:22:38.024142 2044 tablet_bootstrap.cc:654] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:38.027515 1967 leader_election.cc:343] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.018890 1992 tablet_service.cc:1468] Processing CreateTablet for tablet 66ab2bec395d46f2a6a77385c304da93 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250626 08:22:38.037283 1836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a16389239e54c52bb619a62b1b0b91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.042616 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.043304 1928 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.044425 1967 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.045073 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736124bc239f4257b44dfc38e3acc17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.045240 2056 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.045634 2056 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.045989 2056 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:38.047695 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:38.061614 1840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a65a55472e64be4b9ae56426278c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.061867 1986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7d84adc07945fe8a3aa16f71b111e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.063301 1835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9122458a1eae4f07b1a7f4ff63d6be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.066357 1992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66ab2bec395d46f2a6a77385c304da93. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:22:38.052814 1968 leader_election.cc:343] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.054195 1991 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2fc654780843abbda82ba8f5df515e (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250626 08:22:38.075328 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2fc654780843abbda82ba8f5df515e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.075691 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:38.080788 2045 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.081353 2045 ts_tablet_manager.cc:1397] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.255s user 0.018s sys 0.020s
I20250626 08:22:38.101842 2064 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.102373 2064 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.104445 2064 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.106052 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.106072 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.106813 1928 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:38.108042 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.108131 2045 raft_consensus.cc:357] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:38.108685 1816 leader_election.cc:343] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.109025 2045 raft_consensus.cc:383] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.109496 2045 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.108966 2064 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.113392 2064 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.113765 2064 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:38.120048 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 84664us
W20250626 08:22:38.120870 1968 leader_election.cc:343] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.121335 2045 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.122565 2043 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.123028 2043 ts_tablet_manager.cc:1397] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.238s user 0.014s sys 0.015s
I20250626 08:22:38.128427 2056 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.120510 2071 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.120633 1990 tablet_service.cc:1468] Processing CreateTablet for tablet 9122458a1eae4f07b1a7f4ff63d6be19 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250626 08:22:38.125748 2043 raft_consensus.cc:357] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.138701 2043 raft_consensus.cc:383] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.139117 2043 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.097697 1988 tablet_service.cc:1468] Processing CreateTablet for tablet 4a65a55472e64be4b9ae56426278c3c1 (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250626 08:22:38.140079 2043 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.140827 2064 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.139284 2044 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.152082 2044 ts_tablet_manager.cc:1397] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.227s user 0.040s sys 0.027s
I20250626 08:22:38.153411 1988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a65a55472e64be4b9ae56426278c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.155746 2044 raft_consensus.cc:357] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.156733 2044 raft_consensus.cc:383] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.157109 2044 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.158344 1990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9122458a1eae4f07b1a7f4ff63d6be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.158455 1987 tablet_service.cc:1468] Processing CreateTablet for tablet 7a16389239e54c52bb619a62b1b0b91a (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250626 08:22:38.159689 1987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a16389239e54c52bb619a62b1b0b91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.160864 1985 tablet_service.cc:1468] Processing CreateTablet for tablet 5549f2d4a32b4170ad59fc7489d1365f (DEFAULT_TABLE table=test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250626 08:22:38.162499 1985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5549f2d4a32b4170ad59fc7489d1365f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:38.168972 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:38.186348 1927 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.164007 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.171528 2043 ts_tablet_manager.cc:1428] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.048s user 0.007s sys 0.001s
W20250626 08:22:38.189597 1816 leader_election.cc:343] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.163034 2064 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.192327 2043 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.183243 2056 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.168804 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.163908 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:38.208429 1968 leader_election.cc:343] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.212961 2045 ts_tablet_manager.cc:1428] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.131s user 0.042s sys 0.028s
I20250626 08:22:38.220695 1927 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.221841 1967 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:38.223126 2072 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.223632 2072 raft_consensus.cc:695] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:38.224443 2072 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.200219 2044 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.230930 2044 ts_tablet_manager.cc:1428] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.079s user 0.006s sys 0.002s
I20250626 08:22:38.234135 2064 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.234628 2064 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.206226 1928 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.236546 2064 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.238494 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:38.239787 1816 leader_election.cc:343] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.240574 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:22:38.241779 1814 leader_election.cc:343] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.242285 2043 tablet_bootstrap.cc:654] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.242214 1814 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.243567 2064 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.249821 2044 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.256348 2072 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.256851 2072 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.258459 2045 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.257747 1745 catalog_manager.cc:5582] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:38.260499 1928 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:38.261655 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:38.262473 2064 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.262944 2064 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:38.263420 2044 tablet_bootstrap.cc:654] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.264014 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.263795 2064 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.264429 2043 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.264624 1850 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.264909 2043 ts_tablet_manager.cc:1397] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.009s
I20250626 08:22:38.266474 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.267304 2056 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.267727 2043 raft_consensus.cc:357] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.268467 2043 raft_consensus.cc:383] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.268395 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.268818 2043 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.269708 2067 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.270009 2072 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.270179 2067 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.272629 2067 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.274873 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.269465 2043 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.276966 2043 ts_tablet_manager.cc:1428] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
W20250626 08:22:38.277055 1967 leader_election.cc:343] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.277976 2043 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
W20250626 08:22:38.278156 1893 leader_election.cc:343] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.278993 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:38.280349 1893 leader_election.cc:343] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.280776 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.282282 2050 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.267669 2056 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.284240 2056 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.291168 2043 tablet_bootstrap.cc:654] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.292896 2056 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.296379 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.297153 1850 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.299218 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:38.300953 1967 leader_election.cc:343] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.301831 2056 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.307180 1850 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.308682 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:38.309355 2056 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.297128 2045 tablet_bootstrap.cc:654] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.309818 2043 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.310029 2056 raft_consensus.cc:695] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:38.310571 2044 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.310608 2043 ts_tablet_manager.cc:1397] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.002s
I20250626 08:22:38.317389 2044 ts_tablet_manager.cc:1397] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.068s user 0.039s sys 0.015s
I20250626 08:22:38.311108 2056 consensus_queue.cc:237] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.319974 2043 raft_consensus.cc:357] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.320753 2043 raft_consensus.cc:383] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.321102 2043 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
W20250626 08:22:38.311064 1954 tablet.cc:2378] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 08:22:38.322146 2043 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.321017 2044 raft_consensus.cc:357] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.324510 2044 raft_consensus.cc:383] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.324819 2044 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.325809 2045 tablet_bootstrap.cc:492] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.325489 2044 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.326270 2045 ts_tablet_manager.cc:1397] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.009s
I20250626 08:22:38.327997 2044 ts_tablet_manager.cc:1428] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250626 08:22:38.329066 2044 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.330086 2043 ts_tablet_manager.cc:1428] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250626 08:22:38.331209 2043 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.330133 2045 raft_consensus.cc:357] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.334022 2045 raft_consensus.cc:383] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.332954 1744 catalog_manager.cc:5582] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:38.334331 2045 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.335179 2045 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.335988 2044 tablet_bootstrap.cc:654] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.337889 2045 ts_tablet_manager.cc:1428] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250626 08:22:38.337999 2043 tablet_bootstrap.cc:654] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.339253 2045 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.349951 2043 tablet_bootstrap.cc:492] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.350554 2043 ts_tablet_manager.cc:1397] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20250626 08:22:38.353506 2043 raft_consensus.cc:357] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.354503 2043 raft_consensus.cc:383] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.354832 2045 tablet_bootstrap.cc:654] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.355017 2043 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.356361 2043 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.361244 2044 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.361764 2044 ts_tablet_manager.cc:1397] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.014s
I20250626 08:22:38.362761 2043 ts_tablet_manager.cc:1428] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20250626 08:22:38.364557 2044 raft_consensus.cc:357] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.369827 2044 raft_consensus.cc:383] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.370198 2044 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.370719 2043 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.370901 2044 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.366834 2045 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.372068 2045 ts_tablet_manager.cc:1397] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.000s
I20250626 08:22:38.377036 2044 ts_tablet_manager.cc:1428] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250626 08:22:38.378347 2044 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.379738 2045 raft_consensus.cc:357] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.380765 2045 raft_consensus.cc:383] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.381124 2045 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.381572 2043 tablet_bootstrap.cc:654] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.382123 2045 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.384945 2045 ts_tablet_manager.cc:1428] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250626 08:22:38.385118 2044 tablet_bootstrap.cc:654] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.386116 2045 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.392869 2043 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.393342 2043 ts_tablet_manager.cc:1397] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.006s
I20250626 08:22:38.392586 1745 catalog_manager.cc:5582] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:38.395448 2045 tablet_bootstrap.cc:654] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.395840 2043 raft_consensus.cc:357] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.396545 2043 raft_consensus.cc:383] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.396804 2043 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.397610 2043 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.403056 2044 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.403429 2064 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.404052 2064 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.405737 2043 ts_tablet_manager.cc:1428] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250626 08:22:38.406361 2064 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.406692 2043 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.407023 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.407269 2044 ts_tablet_manager.cc:1397] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.012s
I20250626 08:22:38.408874 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:38.410269 1816 leader_election.cc:343] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:22:38.411321 1814 leader_election.cc:343] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.411756 1814 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.411474 2044 raft_consensus.cc:357] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.412231 2043 tablet_bootstrap.cc:654] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.412587 2064 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.413166 2044 raft_consensus.cc:383] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.413520 2044 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.414664 2045 tablet_bootstrap.cc:492] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.414301 2044 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.416944 2044 ts_tablet_manager.cc:1428] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250626 08:22:38.418139 2044 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.415231 2045 ts_tablet_manager.cc:1397] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.004s
I20250626 08:22:38.423589 2045 raft_consensus.cc:357] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.424563 2044 tablet_bootstrap.cc:654] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.424563 2045 raft_consensus.cc:383] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.425129 2045 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.426105 2045 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.429507 2045 ts_tablet_manager.cc:1428] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250626 08:22:38.431289 2056 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.431941 2056 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.436193 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.440186 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.441234 1928 raft_consensus.cc:2391] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:38.442917 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:38.444131 2072 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.452133 2056 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.459980 2043 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.460464 2043 ts_tablet_manager.cc:1397] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.039s
I20250626 08:22:38.460788 2045 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.461138 2072 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.462980 2043 raft_consensus.cc:357] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.463704 2043 raft_consensus.cc:383] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.463999 2043 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.464684 2043 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.466820 2043 ts_tablet_manager.cc:1428] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250626 08:22:38.467198 2045 tablet_bootstrap.cc:654] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.468546 2043 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.474156 2043 tablet_bootstrap.cc:654] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.476158 2044 tablet_bootstrap.cc:492] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.476720 2044 ts_tablet_manager.cc:1397] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.059s user 0.038s sys 0.019s
I20250626 08:22:38.477065 2045 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.477564 2045 ts_tablet_manager.cc:1397] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.000s
I20250626 08:22:38.479383 2044 raft_consensus.cc:357] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.480160 2044 raft_consensus.cc:383] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.480926 2044 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.480309 2045 raft_consensus.cc:357] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.481492 2044 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.481716 2045 raft_consensus.cc:383] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.482187 2045 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.482997 2045 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.484117 2044 ts_tablet_manager.cc:1428] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:38.485092 2044 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.489022 2045 ts_tablet_manager.cc:1428] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250626 08:22:38.490374 2045 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.490836 2044 tablet_bootstrap.cc:654] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.498641 2045 tablet_bootstrap.cc:654] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.501564 2044 tablet_bootstrap.cc:492] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.502027 2044 ts_tablet_manager.cc:1397] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20250626 08:22:38.501972 2067 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.502802 2067 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.504669 2044 raft_consensus.cc:357] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.505369 2067 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.505350 2044 raft_consensus.cc:383] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.506184 2044 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.506814 2044 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.507719 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.508355 1850 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.509310 2044 ts_tablet_manager.cc:1428] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:38.509351 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.510087 2002 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.510360 2044 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.511584 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.513327 2067 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.513820 2067 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.514262 2067 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.516145 2044 tablet_bootstrap.cc:654] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.520908 2043 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.521350 2043 ts_tablet_manager.cc:1397] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.053s user 0.018s sys 0.025s
I20250626 08:22:38.522188 2067 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.523751 2043 raft_consensus.cc:357] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.524405 2043 raft_consensus.cc:383] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.524673 2043 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.524787 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.525315 1850 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.525314 2043 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.531213 2044 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.531795 2044 ts_tablet_manager.cc:1397] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20250626 08:22:38.532114 2072 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.532620 2072 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.534560 2072 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.535470 2043 ts_tablet_manager.cc:1428] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250626 08:22:38.534970 2044 raft_consensus.cc:357] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.536003 2044 raft_consensus.cc:383] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.536389 2044 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.536760 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.537376 1928 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.537245 2044 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.538405 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.538669 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.539036 1849 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.539537 2072 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.539883 2072 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.540154 2044 ts_tablet_manager.cc:1428] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20250626 08:22:38.540203 2072 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.541179 2043 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.548022 2043 tablet_bootstrap.cc:654] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.548650 2044 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.548658 2072 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.550349 1850 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.551818 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.552898 2050 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.554672 2044 tablet_bootstrap.cc:654] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.561102 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.561991 2002 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.570621 2067 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.580902 2050 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:38.582825 2050 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.573684 2002 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.589808 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.590646 1850 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.594322 2099 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.595013 2099 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.598421 2056 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.599056 2056 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.601985 2056 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.603598 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.604425 1928 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.598428 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.606290 1849 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:38.608417 1967 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.611157 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.611968 1849 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.614138 2056 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.614660 2056 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.615098 2056 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.618089 2067 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.618714 2067 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.622305 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.623097 1849 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.624478 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.626454 2101 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.626833 2101 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.627137 2101 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.633231 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:38.633999 1928 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.638172 2072 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.643043 1928 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.644515 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.643965 2101 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.646905 2067 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.648433 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.649042 1849 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.649868 2101 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.650362 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.655503 2064 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.656178 2064 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.658888 2064 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.660354 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.647699 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.668077 2045 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.668751 2045 ts_tablet_manager.cc:1397] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.179s user 0.057s sys 0.073s
I20250626 08:22:38.669936 2072 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.675195 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.677234 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.678241 2064 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.678376 2101 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.672434 2045 raft_consensus.cc:357] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.678844 2064 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.679380 2045 raft_consensus.cc:383] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.679711 2045 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.680347 2045 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.680939 2064 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.683452 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:38.685573 1816 leader_election.cc:343] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.686878 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.687736 1927 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:38.688309 2072 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:38.688127 1742 catalog_manager.cc:5582] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:38.689296 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:38.689648 2072 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:22:38.690785 2048 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.691318 2064 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.691304 2048 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.691838 2064 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.692234 2064 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.692308 2099 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.692790 2105 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.693406 2048 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.693243 2105 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.695113 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.696195 2105 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.696204 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.697052 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.697747 1927 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:38.699064 2056 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.699337 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.699748 2064 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:38.700549 1968 leader_election.cc:343] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.701624 2064 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.685829 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.706920 1928 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:38.707724 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:38.708071 2104 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.708463 1926 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.708642 2104 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.709302 2104 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.710570 1814 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.711700 2064 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.712085 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:38.714867 1850 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.718505 2001 raft_consensus.cc:2391] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:38.718439 2064 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 08:22:38.719357 1814 leader_election.cc:343] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.723052 2101 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.728394 2044 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.728856 2044 ts_tablet_manager.cc:1397] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.180s user 0.040s sys 0.098s
I20250626 08:22:38.729430 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.729673 1926 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:38.730108 2064 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:38.725212 2103 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:38.731139 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.732357 2064 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.732824 2064 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:38.734508 2064 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.735278 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:38.736160 1924 raft_consensus.cc:2391] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:38.737232 1967 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:38.738363 2105 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:38.738329 2044 raft_consensus.cc:357] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.739123 2044 raft_consensus.cc:383] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.739483 2044 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.740108 2044 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.740970 2056 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:22:38.742147 1816 leader_election.cc:343] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.705760 1849 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.742664 1816 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:38.742655 2044 ts_tablet_manager.cc:1428] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250626 08:22:38.744830 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.749897 2048 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 08:22:38.750782 1816 leader_election.cc:343] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.751194 2044 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.746768 2043 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.754271 2043 ts_tablet_manager.cc:1397] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.213s user 0.050s sys 0.067s
I20250626 08:22:38.757086 2043 raft_consensus.cc:357] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.757651 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:38.759276 2099 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.759874 2099 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:38.760350 2056 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:38.765108 1742 catalog_manager.cc:5582] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:38.767555 2044 tablet_bootstrap.cc:654] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.757771 2043 raft_consensus.cc:383] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.769529 2043 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.770153 2043 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.772171 2043 ts_tablet_manager.cc:1428] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250626 08:22:38.772967 2043 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.779774 2043 tablet_bootstrap.cc:654] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.760885 2099 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.790248 1742 catalog_manager.cc:5582] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:38.795181 2045 ts_tablet_manager.cc:1428] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.126s user 0.029s sys 0.048s
I20250626 08:22:38.796525 2045 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.803748 2106 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:38.804327 2045 tablet_bootstrap.cc:654] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.809834 2064 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:38.813760 2044 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.814406 2044 ts_tablet_manager.cc:1397] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.063s user 0.018s sys 0.004s
I20250626 08:22:38.817691 2044 raft_consensus.cc:357] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.818809 2044 raft_consensus.cc:383] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.819104 2044 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.819789 2043 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.820299 2043 ts_tablet_manager.cc:1397] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.000s
I20250626 08:22:38.819876 2044 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.822739 2044 ts_tablet_manager.cc:1428] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250626 08:22:38.823698 2044 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.824230 2045 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.824653 2045 ts_tablet_manager.cc:1397] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.004s
I20250626 08:22:38.826072 2043 raft_consensus.cc:357] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.826728 2043 raft_consensus.cc:383] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.826987 2043 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.827059 2045 raft_consensus.cc:357] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.827800 2045 raft_consensus.cc:383] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.828071 2045 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.829284 2044 tablet_bootstrap.cc:654] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.828859 2045 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.834005 2043 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.834794 2101 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.835219 2101 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.836756 2045 ts_tablet_manager.cc:1428] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250626 08:22:38.838092 2045 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.844132 2101 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.844780 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.845022 2056 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:22:38.846190 1893 leader_election.cc:343] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.846503 2045 tablet_bootstrap.cc:654] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.848294 2043 ts_tablet_manager.cc:1428] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250626 08:22:38.851312 2043 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.857403 2043 tablet_bootstrap.cc:654] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.858975 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.859309 2045 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.860793 2045 ts_tablet_manager.cc:1397] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.009s
I20250626 08:22:38.860831 1850 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.863267 2045 raft_consensus.cc:357] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.863498 1893 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:38.863968 2045 raft_consensus.cc:383] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.864270 2045 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.864172 1746 catalog_manager.cc:5582] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:38.864924 2045 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.865232 2099 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.861754 2044 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.866978 2045 ts_tablet_manager.cc:1428] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:22:38.867220 2044 ts_tablet_manager.cc:1397] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.026s
I20250626 08:22:38.867877 2045 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.869782 2044 raft_consensus.cc:357] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.870468 2044 raft_consensus.cc:383] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.870745 2044 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.874305 2045 tablet_bootstrap.cc:654] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.870483 2101 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.877234 2101 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.878818 2101 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.879079 2043 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.866711 2099 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.879526 2099 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.879527 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.880280 1849 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.882022 2045 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.882509 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.884146 2045 ts_tablet_manager.cc:1397] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.001s
I20250626 08:22:38.884472 2000 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.882083 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.879571 2043 ts_tablet_manager.cc:1397] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.008s
I20250626 08:22:38.886979 2104 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.887864 2101 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.888268 2101 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.889600 2101 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.894852 2043 raft_consensus.cc:357] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.902534 2043 raft_consensus.cc:383] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.902822 2104 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:38.902930 2043 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.895990 2050 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:38.889700 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.897409 2044 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.903887 1850 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.903652 2050 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.903764 2043 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.905632 2050 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.905802 2044 ts_tablet_manager.cc:1428] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20250626 08:22:38.901743 2101 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.895053 2045 raft_consensus.cc:357] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.906517 2045 raft_consensus.cc:383] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:38.906809 2045 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:38.907449 2044 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:38.907491 2045 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.909370 2045 ts_tablet_manager.cc:1428] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.025s user 0.003s sys 0.010s
I20250626 08:22:38.906038 2043 ts_tablet_manager.cc:1428] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20250626 08:22:38.910373 2043 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:38.911877 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:38.912276 2045 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:38.912843 1849 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.913765 2101 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.915515 2099 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.915964 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:38.916288 2044 tablet_bootstrap.cc:654] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.916926 2000 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:38.918354 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.918586 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.919147 1849 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.919268 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:38.919880 1925 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.922060 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.922641 2001 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.924837 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.925210 1850 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.925422 1848 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.926258 2043 tablet_bootstrap.cc:654] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.930896 2099 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.927668 2101 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:38.931017 1968 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.932410 2101 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:38.933571 2101 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.934303 2104 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.934785 2104 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:38.935943 2104 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.936326 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:22:38.937778 1893 leader_election.cc:343] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:38.946213 2045 tablet_bootstrap.cc:654] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:38.947088 2001 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.948810 1925 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:38.948982 1849 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.961604 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:38.965310 2044 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:38.965695 2044 ts_tablet_manager.cc:1397] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.058s user 0.015s sys 0.008s
I20250626 08:22:38.966135 2099 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.968353 2099 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:38.971611 2101 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.968370 1848 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.973273 1893 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:38.973981 2101 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:38.974074 2099 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.975023 2043 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:38.974565 2101 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.975476 2043 ts_tablet_manager.cc:1397] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.017s
I20250626 08:22:38.977147 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:38.977716 2001 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.977931 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:38.978502 1848 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:38.985802 2050 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250626 08:22:38.992405 2001 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:38.978874 2101 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:38.993526 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:38.993717 2101 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.995313 2050 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:38.995793 2050 raft_consensus.cc:695] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:38.996579 2050 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.997974 2045 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:38.998396 2045 ts_tablet_manager.cc:1397] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.006s
I20250626 08:22:38.999202 1848 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.000823 2045 raft_consensus.cc:357] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:38.998667 2043 raft_consensus.cc:357] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.001461 2045 raft_consensus.cc:383] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.002043 2045 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.001677 2043 raft_consensus.cc:383] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.002686 2043 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.001755 1742 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:39.003453 2043 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.004344 2044 raft_consensus.cc:357] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.004977 2044 raft_consensus.cc:383] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.005236 2044 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.006197 2045 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.007063 2044 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.009299 2045 ts_tablet_manager.cc:1428] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20250626 08:22:39.010386 2045 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.011595 2044 ts_tablet_manager.cc:1428] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250626 08:22:39.013991 2044 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.014173 2043 ts_tablet_manager.cc:1428] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20250626 08:22:39.015046 2043 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.020677 2044 tablet_bootstrap.cc:654] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.021092 2043 tablet_bootstrap.cc:654] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.025799 2045 tablet_bootstrap.cc:654] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.030118 2104 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.030637 2104 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.032756 2104 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.033540 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.033849 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.034526 1925 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
W20250626 08:22:39.034812 1968 leader_election.cc:343] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.035781 1967 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:39.032261 2044 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.037235 2104 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.037597 2104 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.037978 2104 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.038405 2044 ts_tablet_manager.cc:1397] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.005s
I20250626 08:22:39.038978 2067 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.040323 2064 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.040804 2064 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.042722 2064 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.043421 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.043881 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.044015 1925 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:39.044447 2000 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:39.044751 2104 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.045121 1814 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:39.045221 2044 raft_consensus.cc:357] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.045962 2044 raft_consensus.cc:383] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.046236 2044 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.046468 2064 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.046833 2064 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.047161 2064 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.049394 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:22:39.052392 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:39.052467 2104 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.047574 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.054073 1925 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.046844 2044 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.055336 1849 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.057523 1744 catalog_manager.cc:5582] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:39.062289 1744 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:39.066366 1744 catalog_manager.cc:5582] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:39.064944 1849 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:39.068895 2064 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.071250 1968 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:39.071856 1925 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:39.072165 2104 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.072619 2104 raft_consensus.cc:695] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:39.073413 2104 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.078429 2044 ts_tablet_manager.cc:1428] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250626 08:22:39.079083 2043 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:39.079653 2044 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.079699 2043 ts_tablet_manager.cc:1397] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.015s
I20250626 08:22:39.082844 2043 raft_consensus.cc:357] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.083482 2043 raft_consensus.cc:383] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.083737 2043 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.084337 2043 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.086340 2043 ts_tablet_manager.cc:1428] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250626 08:22:39.087234 2043 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.087370 2045 tablet_bootstrap.cc:492] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.087819 2045 ts_tablet_manager.cc:1397] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.078s user 0.018s sys 0.007s
I20250626 08:22:39.088627 2064 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.089133 2064 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.086221 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.088336 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.089553 2064 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.090211 2001 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.092139 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:22:39.093470 1814 leader_election.cc:343] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.096136 2043 tablet_bootstrap.cc:654] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.097714 2064 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.098124 1925 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.099197 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.102376 2001 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:22:39.102891 1816 leader_election.cc:343] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.103515 1816 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:39.104462 2064 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.105070 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:39.104956 2045 raft_consensus.cc:357] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.105988 2045 raft_consensus.cc:383] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.106366 2045 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.113337 1925 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:39.114760 2064 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.118974 2064 raft_consensus.cc:695] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:39.120106 2064 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.116792 2044 tablet_bootstrap.cc:654] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.117365 2045 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:39.129285 1816 consensus_peers.cc:487] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Couldn't send request to peer 755d83ca0913406b816e95415d502055. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:39.129837 2045 ts_tablet_manager.cc:1428] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20250626 08:22:39.130540 2104 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.131181 2045 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.131012 2104 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.133818 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.136621 2104 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.137300 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.138280 1925 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:39.139328 1967 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.140483 2104 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.142005 1744 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:39.143465 2044 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.143958 2044 ts_tablet_manager.cc:1397] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.002s
I20250626 08:22:39.147475 2044 raft_consensus.cc:357] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.147881 2043 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:39.148170 2044 raft_consensus.cc:383] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.151036 2044 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.151464 2104 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.149549 2045 tablet_bootstrap.cc:654] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.148409 2043 ts_tablet_manager.cc:1397] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.019s
I20250626 08:22:39.155699 2043 raft_consensus.cc:357] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.156467 2043 raft_consensus.cc:383] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.156813 2043 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.151705 2044 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.157455 2043 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.159669 2043 ts_tablet_manager.cc:1428] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:39.159669 2044 ts_tablet_manager.cc:1428] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250626 08:22:39.160768 2044 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.162261 1744 catalog_manager.cc:5582] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:39.167452 2044 tablet_bootstrap.cc:654] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.171929 2043 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.175766 2045 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.176245 2045 ts_tablet_manager.cc:1397] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.012s
I20250626 08:22:39.178098 2043 tablet_bootstrap.cc:654] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.178961 2045 raft_consensus.cc:357] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.183650 2045 raft_consensus.cc:383] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.183950 2045 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.184568 2045 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.186614 2099 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.187041 2045 ts_tablet_manager.cc:1428] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250626 08:22:39.188030 2044 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.188504 2044 ts_tablet_manager.cc:1397] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.006s
I20250626 08:22:39.190573 2044 raft_consensus.cc:357] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.191210 2044 raft_consensus.cc:383] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.191470 2044 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.192118 2044 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.194080 2045 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.199342 2044 ts_tablet_manager.cc:1428] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 08:22:39.200345 2044 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.204535 2045 tablet_bootstrap.cc:654] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.205780 2104 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.206398 2104 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.208437 2104 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.210038 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:39.211716 1968 leader_election.cc:343] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.213064 2044 tablet_bootstrap.cc:654] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.213310 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.214397 2043 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:39.214870 2043 ts_tablet_manager.cc:1397] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.018s
W20250626 08:22:39.215006 1967 leader_election.cc:343] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.215526 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.216708 2104 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.217803 2043 raft_consensus.cc:357] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.218742 2043 raft_consensus.cc:383] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.218827 2045 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.219266 2043 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.219797 2045 ts_tablet_manager.cc:1397] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.003s
I20250626 08:22:39.220140 2043 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.222889 2045 raft_consensus.cc:357] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.223654 2045 raft_consensus.cc:383] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.223984 2045 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.227900 2044 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.228221 2043 ts_tablet_manager.cc:1428] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20250626 08:22:39.228351 2044 ts_tablet_manager.cc:1397] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.013s
I20250626 08:22:39.229163 2043 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.229409 2045 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.230962 2044 raft_consensus.cc:357] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.231647 2044 raft_consensus.cc:383] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.231913 2044 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.231976 2045 ts_tablet_manager.cc:1428] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250626 08:22:39.232585 2044 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.233074 2045 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.235244 2043 tablet_bootstrap.cc:654] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.236065 2101 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.240782 2045 tablet_bootstrap.cc:654] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.241551 2140 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.252101 2101 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.252594 2101 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.253575 2044 ts_tablet_manager.cc:1428] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20250626 08:22:39.254606 2044 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.256363 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.257009 1849 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.258263 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:39.259074 2067 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.259476 2067 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.259851 2067 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.261188 2044 tablet_bootstrap.cc:654] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.262428 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.263082 2001 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.266047 2045 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.266551 2045 ts_tablet_manager.cc:1397] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.006s
I20250626 08:22:39.270406 2050 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.270869 2050 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.272747 2050 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.273010 2104 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.273227 2045 raft_consensus.cc:357] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.274358 2045 raft_consensus.cc:383] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.274778 2045 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.275858 2045 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.278822 2045 ts_tablet_manager.cc:1428] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250626 08:22:39.280036 2045 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.281350 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.282260 1850 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.289414 2045 tablet_bootstrap.cc:654] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.297760 2104 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.299636 2050 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.300271 2050 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.303580 2050 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.310206 2045 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.311087 2045 ts_tablet_manager.cc:1397] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.016s
I20250626 08:22:39.313058 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.254628 2101 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.315124 2045 raft_consensus.cc:357] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.316181 2045 raft_consensus.cc:383] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.316592 2045 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
W20250626 08:22:39.317258 1893 leader_election.cc:343] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.317474 2045 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.318255 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:39.320600 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.321090 2045 ts_tablet_manager.cc:1428] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250626 08:22:39.321452 1850 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.322573 2045 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.326090 2101 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.326478 2101 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.326836 2101 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.332983 2045 tablet_bootstrap.cc:654] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.334324 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.335208 2001 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.346015 2045 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.347299 2045 ts_tablet_manager.cc:1397] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.011s
I20250626 08:22:39.356606 2045 raft_consensus.cc:357] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.357700 2045 raft_consensus.cc:383] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.358139 2045 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.359536 2105 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.359148 2045 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.364442 2045 ts_tablet_manager.cc:1428] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250626 08:22:39.365790 2045 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.371349 2105 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.372213 2105 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.374974 2105 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.377306 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.378635 2045 tablet_bootstrap.cc:654] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:39.379492 1967 leader_election.cc:343] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.385530 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:39.387243 2050 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.387774 2050 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.388233 2050 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.400712 2045 tablet_bootstrap.cc:492] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.401470 2045 ts_tablet_manager.cc:1397] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.004s
I20250626 08:22:39.402259 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:39.403656 1968 leader_election.cc:343] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.404225 1968 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.404982 2104 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.405134 2045 raft_consensus.cc:357] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.406263 2045 raft_consensus.cc:383] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.406683 2045 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.406946 2104 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.407856 2045 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.411708 2045 ts_tablet_manager.cc:1428] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250626 08:22:39.413164 2045 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.420722 2045 tablet_bootstrap.cc:654] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.421190 2050 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.351276 2067 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.428146 2067 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.429001 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:39.429752 1846 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.439153 1846 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.441175 2067 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.442593 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:39.447970 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.448694 2001 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.451339 2045 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.451794 2045 ts_tablet_manager.cc:1397] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.015s
I20250626 08:22:39.454315 2045 raft_consensus.cc:357] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.454981 2045 raft_consensus.cc:383] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.455336 2045 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.456010 2099 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.456069 2045 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.458829 2045 ts_tablet_manager.cc:1428] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20250626 08:22:39.459918 2045 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.463153 2072 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.463661 2072 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.465682 2072 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.467440 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.467684 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.468398 1850 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:39.470757 2050 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.471799 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.471966 1968 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.472465 2000 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.473073 2104 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.474385 2104 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.475425 2101 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.477368 2101 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.477901 2045 tablet_bootstrap.cc:654] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.486570 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:39.487751 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:39.488337 1845 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.488456 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.489001 2104 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.489081 2002 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.487208 1848 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.492872 2140 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.490028 2001 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.495541 2140 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.490173 2000 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.501765 2140 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.504670 1845 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.505553 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.505939 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:39.507195 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.494940 2099 raft_consensus.cc:695] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:39.508510 1999 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
W20250626 08:22:39.513965 1814 leader_election.cc:343] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.515580 2162 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.516301 1816 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.516877 2162 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.517817 2140 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.520030 2140 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.520483 2140 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.522897 2002 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.524243 2050 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.528337 2162 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.525094 2050 raft_consensus.cc:695] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:39.530659 2050 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.554136 1848 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:39.560755 2099 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.567207 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.569137 2043 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:39.569615 2043 ts_tablet_manager.cc:1397] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.341s user 0.064s sys 0.154s
I20250626 08:22:39.570637 2044 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.571250 2044 ts_tablet_manager.cc:1397] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.317s user 0.061s sys 0.071s
I20250626 08:22:39.572605 2043 raft_consensus.cc:357] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.574164 2043 raft_consensus.cc:383] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.574505 2043 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.575162 2043 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.575788 2044 raft_consensus.cc:357] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.576697 2044 raft_consensus.cc:383] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.577034 2044 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.577535 2043 ts_tablet_manager.cc:1428] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250626 08:22:39.577822 2044 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.578574 2043 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.580964 2044 ts_tablet_manager.cc:1428] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250626 08:22:39.579942 2140 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.582851 2044 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.585528 2104 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.586009 2104 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.589279 2104 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.589525 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:39.590587 2162 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.591010 2162 raft_consensus.cc:695] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:39.591758 2162 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.605486 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.606230 2067 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20250626 08:22:39.608798 1999 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:39.629163 2064 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.629835 2064 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.632345 2064 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.633750 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.634567 1999 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:39.636040 1816 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:39.637564 2043 tablet_bootstrap.cc:654] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.639605 2064 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.640287 2064 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.640617 2064 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.613090 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:39.662361 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.663290 1928 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:39.664983 2048 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:39.665622 2048 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.668170 2048 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:22:39.680594 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 87436.6us
I20250626 08:22:39.679155 2140 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.681393 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:39.686794 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.688683 1928 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:39.688961 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.690058 2002 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:39.691150 1816 leader_election.cc:343] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.693467 2045 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:39.693883 2045 ts_tablet_manager.cc:1397] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.234s user 0.051s sys 0.079s
I20250626 08:22:39.694211 2044 tablet_bootstrap.cc:654] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.696445 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.698165 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:39.698768 1925 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:39.699759 1967 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:22:39.702171 1814 leader_election.cc:343] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.702935 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:39.703840 2105 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.704095 2048 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:39.704213 2105 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.704468 2048 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:39.704589 2105 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.704836 2048 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.706756 2045 raft_consensus.cc:357] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.709513 2045 raft_consensus.cc:383] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.709986 2045 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.710827 2045 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.709095 2072 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20250626 08:22:39.709085 1893 leader_election.cc:343] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.717199 2169 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.718600 2002 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:39.723618 2064 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.726271 2064 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.705381 1745 catalog_manager.cc:5582] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:39.737717 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.738329 1928 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.742885 2045 ts_tablet_manager.cc:1428] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.049s user 0.005s sys 0.003s
I20250626 08:22:39.745278 2169 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.745399 1893 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:39.747274 2162 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:39.748430 2044 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:39.748853 2044 ts_tablet_manager.cc:1397] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.166s user 0.020s sys 0.012s
I20250626 08:22:39.744027 1848 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:39.758708 2064 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:39.765981 2045 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:39.795753 2043 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
W20250626 08:22:39.826877 2172 log.cc:927] Time spent T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20250626 08:22:39.800096 2105 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.826946 1816 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:22:39.835283 2170 log.cc:927] Time spent T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250626 08:22:39.839442 2043 ts_tablet_manager.cc:1397] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.261s user 0.018s sys 0.008s
I20250626 08:22:39.837585 2140 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.847227 2043 raft_consensus.cc:357] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.847980 2043 raft_consensus.cc:383] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.848240 2043 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.848868 2043 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.821156 2162 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20250626 08:22:39.852046 2048 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.856918 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:39.857049 2044 raft_consensus.cc:357] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.857534 2048 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:39.857764 2000 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:39.852027 2043 ts_tablet_manager.cc:1428] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250626 08:22:39.843784 2140 raft_consensus.cc:695] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:39.864326 2043 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:39.864277 2140 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.868639 2105 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:39.857708 2044 raft_consensus.cc:383] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:39.869640 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.870154 2044 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:39.871364 2044 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.886726 2044 ts_tablet_manager.cc:1428] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.138s user 0.007s sys 0.000s
I20250626 08:22:39.860931 1928 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:22:39.879037 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45978us
I20250626 08:22:39.890584 2099 raft_consensus.cc:2762] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250626 08:22:39.910246 2044 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:39.894622 2064 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250626 08:22:39.906708 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:39.924774 1845 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:39.925670 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:22:39.917866 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 64005.4us
I20250626 08:22:39.937121 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:39.937755 1925 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:39.945855 1968 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:39.928850 2048 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:39.946895 2072 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:39.966822 2044 tablet_bootstrap.cc:654] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.967687 1925 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:39.977759 2043 tablet_bootstrap.cc:654] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.961228 2048 raft_consensus.cc:695] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:39.983225 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:22:39.987439 1816 leader_election.cc:343] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:39.988746 2045 tablet_bootstrap.cc:654] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:39.993657 2048 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:39.995129 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:40.006191 2177 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:40.010990 2184 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.011538 2184 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.007493 2177 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:40.017000 2177 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.017643 2064 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.018025 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:40.021080 1968 leader_election.cc:343] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.016894 2184 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:40.026491 2050 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.029597 2050 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.031664 2050 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.033666 2171 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:40.032413 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:40.018117 2064 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.052088 2044 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:40.052597 2044 ts_tablet_manager.cc:1397] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.143s user 0.012s sys 0.021s
I20250626 08:22:40.052412 2064 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.058369 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:40.062907 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:40.064031 1967 leader_election.cc:343] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.064119 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:40.064414 1816 leader_election.cc:343] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.064543 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:22:40.063072 1893 leader_election.cc:343] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.064849 1999 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:40.072201 2185 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:40.066757 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:40.075331 2171 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:40.075852 2171 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:40.076285 2171 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.065840 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:40.095279 2043 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:40.097148 2043 ts_tablet_manager.cc:1397] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.233s user 0.013s sys 0.009s
I20250626 08:22:40.107638 2101 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20250626 08:22:40.114686 1814 leader_election.cc:343] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.115207 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:40.117972 2064 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:40.165347 2162 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250626 08:22:40.167420 2044 raft_consensus.cc:357] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.165055 2050 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250626 08:22:40.169725 2043 raft_consensus.cc:357] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.170461 2043 raft_consensus.cc:383] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.170742 2043 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.171324 2043 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.172866 2045 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:40.173305 2045 ts_tablet_manager.cc:1397] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.408s user 0.019s sys 0.023s
I20250626 08:22:40.173496 2043 ts_tablet_manager.cc:1428] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.076s user 0.004s sys 0.000s
I20250626 08:22:40.168180 2044 raft_consensus.cc:383] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.174156 2044 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.174983 2044 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.178692 2043 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:40.183446 2045 raft_consensus.cc:357] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.185458 2045 raft_consensus.cc:383] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.185738 2045 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.186396 2045 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.188097 2045 ts_tablet_manager.cc:1428] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250626 08:22:40.183338 2067 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250626 08:22:40.201479 2044 ts_tablet_manager.cc:1428] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.146s user 0.018s sys 0.023s
I20250626 08:22:40.202966 2044 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:40.236382 2044 tablet_bootstrap.cc:654] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.239805 2171 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.242322 2171 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.255627 2045 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:40.271158 1745 catalog_manager.cc:5582] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:40.273005 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:40.273520 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:40.274343 1999 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:40.275269 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20250626 08:22:40.277796 1893 leader_election.cc:343] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.279424 2044 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:40.279853 2044 ts_tablet_manager.cc:1397] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.001s
I20250626 08:22:40.257411 2043 tablet_bootstrap.cc:654] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.301177 2044 raft_consensus.cc:357] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.302016 2044 raft_consensus.cc:383] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.302336 2044 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.305836 1746 catalog_manager.cc:5582] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:22:40.306962 1999 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:40.307722 2045 tablet_bootstrap.cc:654] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.308149 2044 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.307780 1746 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:22:40.310963 2171 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20250626 08:22:40.355201 2183 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.367486 2183 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.326323 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:22:40.379716 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 53543.9us
I20250626 08:22:40.380321 2072 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.380941 2072 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.382083 2101 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:40.382560 2101 raft_consensus.cc:695] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:40.383013 2044 ts_tablet_manager.cc:1428] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.103s user 0.009s sys 0.012s
I20250626 08:22:40.383363 2101 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.383944 2044 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:40.393117 2072 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:40.397683 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:40.398586 2064 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:40.413061 2001 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.417660 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:40.422456 1926 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:40.464493 1967 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:40.533702 2183 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.623349 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:40.624285 1998 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:40.668229 1745 catalog_manager.cc:5582] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
W20250626 08:22:40.674089 2172 log.cc:927] Time spent T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.263s user 0.108s sys 0.089s
I20250626 08:22:40.674455 2043 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:40.674798 1816 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:40.675064 2043 ts_tablet_manager.cc:1397] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.497s user 0.012s sys 0.009s
W20250626 08:22:40.676807 2168 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.252s user 0.027s sys 0.023s
I20250626 08:22:40.676499 1745 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:40.680794 2043 raft_consensus.cc:357] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.681624 2043 raft_consensus.cc:383] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.682093 2043 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.683163 2177 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:40.683555 2177 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:40.683908 2177 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.684206 2064 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.684691 2064 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.686657 2064 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.690888 2044 tablet_bootstrap.cc:654] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.690963 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:40.692504 2043 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.693645 1998 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:40.694276 2043 ts_tablet_manager.cc:1428] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20250626 08:22:40.695231 2043 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:40.696004 2045 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:40.696465 2045 ts_tablet_manager.cc:1397] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.441s user 0.049s sys 0.053s
W20250626 08:22:40.680557 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67085us
I20250626 08:22:40.700398 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:40.700973 1816 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:40.703435 2045 raft_consensus.cc:357] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.704124 2045 raft_consensus.cc:383] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.704398 2045 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.704746 1926 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.705093 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:40.705031 2045 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.707587 2045 ts_tablet_manager.cc:1428] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250626 08:22:40.708806 2183 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:40.711241 2183 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:40.711656 2183 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:40.709054 1814 leader_election.cc:343] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.702517 2177 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.711892 2045 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:40.713110 2072 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:40.715210 2179 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250626 08:22:40.718490 2179 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:40.721362 2064 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:40.725070 2177 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:22:40.744108 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 279762us
W20250626 08:22:40.742044 1814 leader_election.cc:343] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.758432 1746 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:40.762599 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:40.763192 2000 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.756814 2177 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250626 08:22:40.765050 2101 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.765507 2101 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.777943 2044 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:40.778424 2044 ts_tablet_manager.cc:1397] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.395s user 0.016s sys 0.012s
I20250626 08:22:40.769855 2000 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:40.781739 1816 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:40.782873 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:40.783453 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:40.784359 2045 tablet_bootstrap.cc:654] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.784538 2177 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:40.784977 2177 raft_consensus.cc:695] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:40.786103 1893 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:40.786320 2177 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.787217 2169 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:22:40.789170 2064 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:40.769368 2101 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.803433 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:40.818274 1814 leader_election.cc:343] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.818169 2171 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:40.834308 2043 tablet_bootstrap.cc:654] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.841352 2044 raft_consensus.cc:357] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.842453 2044 raft_consensus.cc:383] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.842737 2044 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.845721 2183 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.853067 2044 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.858759 2044 ts_tablet_manager.cc:1428] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.080s user 0.007s sys 0.001s
I20250626 08:22:40.859695 2044 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:40.860178 2183 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.861142 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:40.861793 1999 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:40.862741 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:40.863978 1814 leader_election.cc:343] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:40.877022 2045 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:40.877516 2045 ts_tablet_manager.cc:1397] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.166s user 0.010s sys 0.010s
W20250626 08:22:40.878419 1893 consensus_peers.cc:487] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:40.879962 2044 tablet_bootstrap.cc:654] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.893931 1999 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:40.895164 1816 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:40.883175 2045 raft_consensus.cc:357] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.896698 2045 raft_consensus.cc:383] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.896905 2183 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:40.897054 2045 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.897565 2183 raft_consensus.cc:695] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:40.898106 2045 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.898432 2183 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.900501 2045 ts_tablet_manager.cc:1428] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.023s user 0.002s sys 0.005s
I20250626 08:22:40.901522 2045 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:40.905418 2044 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:40.905817 2044 ts_tablet_manager.cc:1397] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.004s
I20250626 08:22:40.908223 2043 tablet_bootstrap.cc:492] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:40.908725 2043 ts_tablet_manager.cc:1397] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.214s user 0.006s sys 0.010s
I20250626 08:22:40.908986 2044 raft_consensus.cc:357] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.909503 2044 raft_consensus.cc:383] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.909726 2044 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.910282 2044 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.911322 2043 raft_consensus.cc:357] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.912015 2043 raft_consensus.cc:383] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.912287 2043 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.912943 2043 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.915236 2043 ts_tablet_manager.cc:1428] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250626 08:22:40.916090 2043 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:40.916621 2044 ts_tablet_manager.cc:1428] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 08:22:40.917526 2044 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:40.922104 2043 tablet_bootstrap.cc:654] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.922590 2045 tablet_bootstrap.cc:654] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.928413 2043 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:40.928893 2043 ts_tablet_manager.cc:1397] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20250626 08:22:40.931315 2043 raft_consensus.cc:357] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.933776 2045 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:40.934082 2043 raft_consensus.cc:383] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.934360 2045 ts_tablet_manager.cc:1397] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.004s
I20250626 08:22:40.934373 2043 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.935133 2044 tablet_bootstrap.cc:654] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.935271 2043 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.937186 2045 raft_consensus.cc:357] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.938876 2045 raft_consensus.cc:383] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.939085 2045 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.939383 2043 ts_tablet_manager.cc:1428] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250626 08:22:40.939640 2045 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.940757 2043 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:40.942445 2045 ts_tablet_manager.cc:1428] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250626 08:22:40.943886 2045 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:40.962528 2043 tablet_bootstrap.cc:654] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.972996 2044 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:40.973489 2044 ts_tablet_manager.cc:1397] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.056s user 0.021s sys 0.010s
I20250626 08:22:40.973316 1745 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:40.975158 1745 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:40.976022 2044 raft_consensus.cc:357] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.976678 2044 raft_consensus.cc:383] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.976953 2044 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.977573 2044 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.978399 2045 tablet_bootstrap.cc:654] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:40.984205 2043 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:40.984642 2043 ts_tablet_manager.cc:1397] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.009s
I20250626 08:22:40.986783 2043 raft_consensus.cc:357] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.987461 2043 raft_consensus.cc:383] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.987795 2045 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:40.987727 2043 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.988270 2045 ts_tablet_manager.cc:1397] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.005s
I20250626 08:22:40.988476 2043 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.990794 2045 raft_consensus.cc:357] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.991453 2045 raft_consensus.cc:383] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:40.991581 2044 ts_tablet_manager.cc:1428] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250626 08:22:40.991739 2045 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:40.992659 2044 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:40.993239 2183 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:40.993685 2183 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.995685 2183 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:40.997548 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:40.992503 2045 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:40.998679 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.000008 2043 ts_tablet_manager.cc:1428] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
W20250626 08:22:41.000115 1814 leader_election.cc:343] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.000952 2044 tablet_bootstrap.cc:654] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.001271 2043 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.002229 2045 ts_tablet_manager.cc:1428] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
W20250626 08:22:41.003907 1816 leader_election.cc:343] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.004400 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:41.005141 2045 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.005192 2183 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.008795 2043 tablet_bootstrap.cc:654] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.023058 2045 tablet_bootstrap.cc:654] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.026105 2044 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.026631 2044 ts_tablet_manager.cc:1397] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.009s
I20250626 08:22:41.036120 2044 raft_consensus.cc:357] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.036860 2044 raft_consensus.cc:383] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.037147 2044 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.037776 2044 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.044440 2044 ts_tablet_manager.cc:1428] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.017s user 0.002s sys 0.004s
I20250626 08:22:41.045110 2043 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.045578 2044 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.045697 2043 ts_tablet_manager.cc:1397] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.001s
I20250626 08:22:41.048593 2043 raft_consensus.cc:357] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.049810 2101 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.050328 2101 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.052300 2101 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.054129 2045 tablet_bootstrap.cc:492] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.054108 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.054345 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.054795 2045 ts_tablet_manager.cc:1397] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
I20250626 08:22:41.055008 1999 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
W20250626 08:22:41.055629 1893 leader_election.cc:343] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.049398 2043 raft_consensus.cc:383] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.056461 2044 tablet_bootstrap.cc:654] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.056357 1893 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:41.057152 2101 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.057274 2045 raft_consensus.cc:357] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.056473 2043 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.060963 2045 raft_consensus.cc:383] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.060985 2101 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.061503 2045 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.061648 2101 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.062248 2045 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.064518 2045 ts_tablet_manager.cc:1428] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 08:22:41.065485 2045 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.070158 2101 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.065234 2043 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.071468 2045 tablet_bootstrap.cc:654] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.079761 2043 ts_tablet_manager.cc:1428] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.034s user 0.004s sys 0.001s
I20250626 08:22:41.081050 2043 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.091430 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:22:41.093232 1893 leader_election.cc:343] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.094928 2101 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.096218 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.097011 1999 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.098879 2044 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.099359 2044 ts_tablet_manager.cc:1397] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.019s
I20250626 08:22:41.102346 2044 raft_consensus.cc:357] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.102865 1999 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:41.103078 2044 raft_consensus.cc:383] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.103395 2044 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.104113 1893 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:41.104144 2044 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.105276 2045 tablet_bootstrap.cc:492] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.105451 2101 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.105757 2045 ts_tablet_manager.cc:1397] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20250626 08:22:41.106212 2101 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:41.107163 2101 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.108361 2045 raft_consensus.cc:357] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.109041 2045 raft_consensus.cc:383] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.109305 2045 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.109942 2045 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.112183 2045 ts_tablet_manager.cc:1428] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:22:41.113085 2045 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.116161 2044 ts_tablet_manager.cc:1428] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250626 08:22:41.117005 2044 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.119117 2171 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.119796 2171 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.124020 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.124082 2171 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.124096 2045 tablet_bootstrap.cc:654] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.126780 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.128201 1999 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:41.129406 2044 tablet_bootstrap.cc:654] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.129751 1893 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:22:41.135069 1893 leader_election.cc:343] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.136586 2067 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.137061 2067 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.137492 2067 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.140544 2043 tablet_bootstrap.cc:654] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.151688 2169 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.152566 2169 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.161415 1746 catalog_manager.cc:5582] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:41.164224 2169 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.163959 2067 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.162241 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.165196 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.165503 1850 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:41.165812 1999 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:41.168653 2067 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.169327 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.171212 1893 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.172596 2043 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.173040 2043 ts_tablet_manager.cc:1397] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.009s
I20250626 08:22:41.173115 2101 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.173484 2101 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.174355 2101 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.174997 2105 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.175460 2105 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.178426 2105 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.180162 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.180307 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.180716 2043 raft_consensus.cc:357] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.180819 1927 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.181469 2043 raft_consensus.cc:383] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.180935 2183 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.181783 2043 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.182045 2183 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.182510 2043 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.183997 2183 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.184695 2043 ts_tablet_manager.cc:1428] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
W20250626 08:22:41.177042 1893 leader_election.cc:343] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.186146 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.186508 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.186699 1999 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.186615 2101 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.187227 2001 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:41.188545 2101 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.189191 2169 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.189656 2169 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.190623 2044 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.191061 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.191100 2044 ts_tablet_manager.cc:1397] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.010s
I20250626 08:22:41.191803 2105 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.192054 2169 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.192622 2043 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.192334 2105 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.193471 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.193591 2105 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.194780 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.194701 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.193496 1816 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.195410 1926 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:41.195684 1850 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.197330 1848 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.198691 2064 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.199052 2064 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.203691 2169 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.204370 2169 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.199363 2064 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.208361 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.197137 2001 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.209357 1845 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:41.212344 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.213732 2169 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.219686 1999 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:41.221143 2072 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.221630 2072 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.222131 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.224797 2184 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.225244 2184 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.225854 2072 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.223346 1845 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:41.227648 2045 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.228094 2045 ts_tablet_manager.cc:1397] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.015s
I20250626 08:22:41.230243 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.230396 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.231027 1926 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.228713 2184 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:22:41.231357 1968 leader_election.cc:343] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:22:41.231024 1814 consensus_peers.cc:487] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Couldn't send request to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:41.229394 2044 raft_consensus.cc:357] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.232250 2044 raft_consensus.cc:383] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.232223 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.232482 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.232671 2044 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.224196 1848 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:41.233494 2044 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:41.234107 1967 leader_election.cc:343] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.245329 1967 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:41.246129 2072 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.234663 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.233327 2184 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.248056 2044 ts_tablet_manager.cc:1428] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.056s user 0.005s sys 0.001s
I20250626 08:22:41.248775 2184 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.249157 2184 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.249650 2044 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.248093 1850 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.235144 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.235329 2064 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.224398 2001 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
W20250626 08:22:41.236485 1893 leader_election.cc:343] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.252512 1893 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:41.253341 2067 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.253829 2067 raft_consensus.cc:695] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:41.253993 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:41.254601 1927 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.254734 2067 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.255957 2184 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.259732 1893 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.260393 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.260609 2169 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.261032 1850 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.261138 2169 raft_consensus.cc:695] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:41.262070 2169 consensus_queue.cc:237] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.264376 2177 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.264842 2177 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.265879 1893 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:41.266525 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.266822 2177 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.267398 2171 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.267800 2171 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.268108 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.268383 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.269026 2000 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:41.268133 2171 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.238871 2043 tablet_bootstrap.cc:654] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.240384 2045 raft_consensus.cc:357] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.274314 2045 raft_consensus.cc:383] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.274626 2045 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.274914 1816 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:41.275285 2045 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.276988 2177 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.277583 2045 ts_tablet_manager.cc:1428] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.049s user 0.002s sys 0.001s
I20250626 08:22:41.263907 1999 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:41.241766 2105 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.286463 2045 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.265601 2064 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.287635 1927 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:41.289579 2001 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:41.293519 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.293967 1814 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:41.294369 1848 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:41.294667 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:41.295202 2064 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.295420 1926 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:41.295867 2064 raft_consensus.cc:695] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:41.296660 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:41.297500 2072 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.281842 2171 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.298416 2105 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.305656 2171 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.313422 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:41.298818 2064 consensus_queue.cc:237] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.315425 2101 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.315891 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.316218 2101 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.316537 1848 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.316615 2101 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.316962 2184 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.319288 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:41.319725 1850 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:41.322530 2045 tablet_bootstrap.cc:654] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.322584 1968 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.309626 2177 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.324510 2184 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.325202 2184 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:41.326246 2184 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.329476 2043 tablet_bootstrap.cc:492] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.330191 2043 ts_tablet_manager.cc:1397] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.139s user 0.014s sys 0.002s
I20250626 08:22:41.336949 2044 tablet_bootstrap.cc:654] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.343132 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.360096 1848 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
W20250626 08:22:41.363302 1893 leader_election.cc:343] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.366089 1893 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:41.367983 2067 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.362310 2101 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.369036 2067 raft_consensus.cc:695] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:41.370252 1927 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
W20250626 08:22:41.370968 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:41.370419 2067 consensus_queue.cc:237] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.379698 1745 catalog_manager.cc:5582] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:41.382575 1746 catalog_manager.cc:5582] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:41.385305 2043 raft_consensus.cc:357] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.385810 2043 raft_consensus.cc:383] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.386441 2043 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.391996 2043 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.394829 2101 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.398515 1744 catalog_manager.cc:5582] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:41.403184 2043 ts_tablet_manager.cc:1428] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.073s user 0.006s sys 0.001s
I20250626 08:22:41.404441 2043 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.405311 2045 tablet_bootstrap.cc:492] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.405961 2045 ts_tablet_manager.cc:1397] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.120s user 0.022s sys 0.013s
I20250626 08:22:41.399353 2064 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250626 08:22:41.420686 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:41.420841 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.421073 1848 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.421634 1850 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:41.436329 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:41.441401 2101 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.450217 2105 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.450255 2244 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:41.450897 2105 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.452877 2105 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.454478 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.455142 1850 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.456271 2044 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.456732 2044 ts_tablet_manager.cc:1397] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.207s user 0.010s sys 0.005s
I20250626 08:22:41.456578 1968 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.465014 2044 raft_consensus.cc:357] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.462865 2105 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.466266 2105 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.466383 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.466722 2105 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.467069 1927 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:41.465744 2044 raft_consensus.cc:383] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.467993 2044 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.470809 2044 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.483122 2045 raft_consensus.cc:357] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.484162 2045 raft_consensus.cc:383] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.484493 2045 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.485193 2045 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.486269 2043 tablet_bootstrap.cc:654] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.509516 2044 ts_tablet_manager.cc:1428] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.052s user 0.011s sys 0.009s
I20250626 08:22:41.510334 2045 ts_tablet_manager.cc:1428] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.104s user 0.003s sys 0.005s
I20250626 08:22:41.512195 2045 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Bootstrap starting.
W20250626 08:22:41.514149 1814 consensus_peers.cc:487] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Couldn't send request to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:41.514878 1850 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.517832 2044 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.501344 1746 catalog_manager.cc:5582] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:41.523381 1746 catalog_manager.cc:5582] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:41.540627 2169 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.543722 2043 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.544656 2043 ts_tablet_manager.cc:1397] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.141s user 0.013s sys 0.007s
I20250626 08:22:41.547545 2105 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.549103 2044 tablet_bootstrap.cc:654] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.551532 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:41.552150 1848 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.554401 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:41.555015 1927 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.552718 2105 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.561877 1848 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:41.564930 1927 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:41.566943 2045 tablet_bootstrap.cc:654] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.571592 1967 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:41.573426 2043 raft_consensus.cc:357] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.581102 2043 raft_consensus.cc:383] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.581395 2043 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.582007 2043 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.584163 2043 ts_tablet_manager.cc:1428] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250626 08:22:41.573890 2105 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:41.585153 2105 raft_consensus.cc:695] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:41.586217 2105 consensus_queue.cc:237] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.589670 2043 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.608839 2043 tablet_bootstrap.cc:654] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.636468 2044 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.637006 2044 ts_tablet_manager.cc:1397] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.126s user 0.032s sys 0.025s
I20250626 08:22:41.639905 2044 raft_consensus.cc:357] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.640663 2044 raft_consensus.cc:383] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.640761 2238 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.641062 2044 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.643122 2045 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.641942 2044 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.643857 1746 catalog_manager.cc:5582] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:41.644802 2043 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.647397 2043 ts_tablet_manager.cc:1397] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.006s
I20250626 08:22:41.649708 2044 ts_tablet_manager.cc:1428] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250626 08:22:41.649993 2043 raft_consensus.cc:357] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.650637 2043 raft_consensus.cc:383] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.650866 2043 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.651409 2043 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.653470 2043 ts_tablet_manager.cc:1428] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250626 08:22:41.645191 2045 ts_tablet_manager.cc:1397] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.014s
I20250626 08:22:41.656241 2045 raft_consensus.cc:357] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.656877 2045 raft_consensus.cc:383] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.657150 2045 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.657773 2045 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.659965 2045 ts_tablet_manager.cc:1428] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:41.660881 2045 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.650637 2044 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.663722 2238 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.664219 2238 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.666365 2238 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.667510 2044 tablet_bootstrap.cc:654] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.667841 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.667527 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.670334 2043 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.671331 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:41.672351 2238 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.675366 2043 tablet_bootstrap.cc:654] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.680677 2169 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.685037 2045 tablet_bootstrap.cc:654] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.690135 2044 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.690519 2044 ts_tablet_manager.cc:1397] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.007s
I20250626 08:22:41.693111 2044 raft_consensus.cc:357] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.693835 2044 raft_consensus.cc:383] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.694172 2044 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.694868 2044 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.697659 2044 ts_tablet_manager.cc:1428] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:41.698952 2044 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.711338 2045 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.711845 2045 ts_tablet_manager.cc:1397] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.004s
I20250626 08:22:41.712004 2169 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.719884 2043 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.721534 2043 ts_tablet_manager.cc:1397] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.003s
I20250626 08:22:41.720274 2045 raft_consensus.cc:357] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.722069 2045 raft_consensus.cc:383] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.722375 2045 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.723028 2045 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.724866 2044 tablet_bootstrap.cc:654] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.726027 2238 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.727295 2043 raft_consensus.cc:357] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.728302 2043 raft_consensus.cc:383] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.729467 2043 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.730232 2043 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.730270 2105 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:41.734107 2043 ts_tablet_manager.cc:1428] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250626 08:22:41.735266 2043 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.740990 2045 ts_tablet_manager.cc:1428] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20250626 08:22:41.741955 2045 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.747720 2045 tablet_bootstrap.cc:654] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.750749 2044 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.751375 2044 ts_tablet_manager.cc:1397] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.001s
I20250626 08:22:41.753947 2044 raft_consensus.cc:357] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.754601 2044 raft_consensus.cc:383] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.754890 2044 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.755468 2044 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.760596 2044 ts_tablet_manager.cc:1428] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250626 08:22:41.760684 2184 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.760655 2043 tablet_bootstrap.cc:654] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.761559 2044 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.761240 2184 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.763178 2184 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:41.764708 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:41.767438 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.769093 2044 tablet_bootstrap.cc:654] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:41.769246 1967 leader_election.cc:343] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.771965 2045 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:41.772444 2045 ts_tablet_manager.cc:1397] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
W20250626 08:22:41.774331 1968 leader_election.cc:343] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.774997 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:41.776798 2105 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.775069 2045 raft_consensus.cc:357] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.780493 2045 raft_consensus.cc:383] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.780824 2045 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.781106 2064 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.781459 2045 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.781580 2064 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.783095 2043 tablet_bootstrap.cc:492] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:41.783464 2043 ts_tablet_manager.cc:1397] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.000s
I20250626 08:22:41.784044 2064 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.785246 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.786074 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:41.786655 1814 leader_election.cc:343] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.787381 2045 ts_tablet_manager.cc:1428] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20250626 08:22:41.788286 2045 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:41.788699 2101 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.793222 1999 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.794374 2043 raft_consensus.cc:357] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.795068 2043 raft_consensus.cc:383] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.795325 2043 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.795902 2043 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:41.788175 1816 leader_election.cc:343] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.797995 2043 ts_tablet_manager.cc:1428] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250626 08:22:41.798156 1816 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:41.798849 2043 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:41.801777 2177 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:41.806435 2045 tablet_bootstrap.cc:654] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.807591 2184 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:41.813719 1926 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:41.820093 2265 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250626 08:22:41.823107 2171 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:41.862162 2044 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:41.938457 2044 ts_tablet_manager.cc:1397] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.177s user 0.017s sys 0.037s
I20250626 08:22:41.942878 2183 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:41.948949 2183 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.950873 2183 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:41.956424 2043 tablet_bootstrap.cc:654] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:41.956619 2044 raft_consensus.cc:357] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.957286 2044 raft_consensus.cc:383] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:41.957577 2044 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:41.958877 2044 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:41.963692 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:41.965178 2177 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:41.967862 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:41.968742 2000 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:41.969022 2044 ts_tablet_manager.cc:1428] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20250626 08:22:41.970050 2044 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:41.970305 2183 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:22:41.973002 1814 leader_election.cc:343] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:41.974606 1927 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:41.980337 1816 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:41.981630 2183 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:41.982057 2183 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:41.982391 2183 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.002964 2183 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.005371 2171 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:42.006103 2183 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.008478 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.009092 2000 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.011943 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:42.015034 1814 leader_election.cc:343] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.015718 2000 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:42.016637 1816 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.017407 2244 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.017858 2244 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:42.018668 2244 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.026142 2171 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.026669 2171 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.028590 2171 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.037875 2045 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.038406 2045 ts_tablet_manager.cc:1397] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.250s user 0.018s sys 0.012s
I20250626 08:22:42.040516 2045 raft_consensus.cc:357] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.040995 2045 raft_consensus.cc:383] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.041220 2045 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.049671 2044 tablet_bootstrap.cc:654] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.053603 2105 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.054121 2105 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.056010 2105 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.058497 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.058768 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.059305 1927 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.060508 1967 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:22:42.061123 1968 leader_election.cc:343] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.061710 2105 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.062028 2105 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.062286 2105 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.060509 2045 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.067355 2045 ts_tablet_manager.cc:1428] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250626 08:22:42.067811 2244 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.068522 2045 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.068356 2244 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.071580 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.072520 2244 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.072789 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:42.073055 1814 leader_election.cc:343] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.073351 2000 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:42.074431 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.075209 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.075731 2244 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.076148 2244 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.076548 2244 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.078264 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.084398 2244 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:42.085144 1893 leader_election.cc:343] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.087292 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:42.088546 1814 leader_election.cc:343] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.089565 2244 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.090740 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.091308 1999 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.094516 2105 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.096971 2000 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:42.098547 1999 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:42.100001 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.100735 2244 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.101235 2244 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:42.101233 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.102401 2244 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:42.102929 1968 leader_election.cc:343] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.111141 1742 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:42.118381 2105 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.118623 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:42.119196 1927 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.126411 1927 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:42.127743 1967 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:42.128396 2043 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.129063 2043 ts_tablet_manager.cc:1397] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.330s user 0.009s sys 0.026s
I20250626 08:22:42.129331 2105 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.130326 2105 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:42.131923 2105 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.133077 2043 raft_consensus.cc:357] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.134474 2043 raft_consensus.cc:383] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.134897 2043 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.140668 2177 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.142140 2177 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.239269 2169 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.239817 2169 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.252977 2045 tablet_bootstrap.cc:700] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Time spent opening tablet: real 0.104s user 0.006s sys 0.003s
I20250626 08:22:42.253779 2045 tablet_bootstrap.cc:654] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.291100 2043 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.298004 2184 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20250626 08:22:42.314697 2044 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.315184 2044 ts_tablet_manager.cc:1397] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.345s user 0.033s sys 0.075s
I20250626 08:22:42.296854 1742 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:42.316130 1893 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.314098 2177 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.318714 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.319432 1844 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:42.320989 2169 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.324296 2288 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.324798 2288 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.326815 2288 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.333490 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.332906 2044 raft_consensus.cc:357] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.334940 2044 raft_consensus.cc:383] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.335232 2044 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.337414 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:42.337384 2044 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.344558 2105 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:42.345275 2238 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.345643 2238 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.345986 2238 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.344944 2044 ts_tablet_manager.cc:1428] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.029s user 0.003s sys 0.002s
I20250626 08:22:42.335244 2290 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.340137 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.341441 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.342164 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.351451 1848 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.342478 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.351594 1998 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:42.350178 2044 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.353745 2043 ts_tablet_manager.cc:1428] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.224s user 0.013s sys 0.008s
I20250626 08:22:42.354730 2043 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.356088 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.356923 2184 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.358296 2184 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.358675 2184 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.359526 2238 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.360687 1927 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:42.361404 2238 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.363153 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.365702 2177 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.366215 2177 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.366638 2177 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.379453 1745 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:42.383855 2177 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.385860 2177 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:22:42.392222 1967 leader_election.cc:343] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.394809 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.395620 1998 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.400095 2044 tablet_bootstrap.cc:654] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.406311 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:42.407130 1924 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:42.410171 2184 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.414251 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.414466 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.415095 1848 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:42.416079 2045 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.417055 2045 ts_tablet_manager.cc:1397] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.349s user 0.018s sys 0.023s
I20250626 08:22:42.418440 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.416519 2184 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.422806 2290 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.424048 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.424732 1848 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.429747 1848 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:42.430864 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.431711 2105 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.432154 2105 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:42.432495 2043 tablet_bootstrap.cc:654] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.423305 1998 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:42.436874 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.437968 2177 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.438419 2177 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:42.438308 2105 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.439311 2177 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.441176 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:42.442714 1967 leader_election.cc:343] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.447227 2045 raft_consensus.cc:357] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.447983 2045 raft_consensus.cc:383] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.448256 2045 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.449946 2045 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.452194 2045 ts_tablet_manager.cc:1428] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.035s user 0.004s sys 0.000s
I20250626 08:22:42.453100 2045 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
W20250626 08:22:42.462071 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:42.476141 1742 catalog_manager.cc:5582] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:42.478571 2045 tablet_bootstrap.cc:654] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.480011 1744 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:42.490335 2043 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.490828 2043 ts_tablet_manager.cc:1397] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.136s user 0.009s sys 0.007s
I20250626 08:22:42.493167 2043 raft_consensus.cc:357] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.493652 2043 raft_consensus.cc:383] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.493873 2043 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.494460 2043 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.495915 2043 ts_tablet_manager.cc:1428] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20250626 08:22:42.497098 2043 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.501786 2044 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.502436 2044 ts_tablet_manager.cc:1397] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.152s user 0.018s sys 0.003s
I20250626 08:22:42.502897 2177 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:42.513348 2043 tablet_bootstrap.cc:654] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.518088 2286 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:42.526543 2279 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.527253 2279 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.531069 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.532100 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.533824 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.534626 2290 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.526115 2044 raft_consensus.cc:357] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.535815 2044 raft_consensus.cc:383] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.534580 2279 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.536211 2044 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.539636 2044 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.551046 2044 ts_tablet_manager.cc:1428] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.048s user 0.005s sys 0.002s
I20250626 08:22:42.551213 2045 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.551658 2045 ts_tablet_manager.cc:1397] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.099s user 0.034s sys 0.033s
I20250626 08:22:42.551983 2044 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.554050 2045 raft_consensus.cc:357] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.554750 2045 raft_consensus.cc:383] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.555038 2045 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.555764 2045 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.559378 2045 ts_tablet_manager.cc:1428] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250626 08:22:42.560297 2045 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.561367 2044 tablet_bootstrap.cc:654] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.566864 2045 tablet_bootstrap.cc:654] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.573266 1927 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.595906 2044 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.596582 2044 ts_tablet_manager.cc:1397] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.000s
I20250626 08:22:42.597069 2286 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:42.602511 2044 raft_consensus.cc:357] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.603202 2044 raft_consensus.cc:383] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.603458 2044 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.604033 2044 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.620682 2045 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.621007 2044 ts_tablet_manager.cc:1428] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20250626 08:22:42.621153 2045 ts_tablet_manager.cc:1397] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.061s user 0.028s sys 0.024s
I20250626 08:22:42.621544 2043 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.622020 2044 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.622046 2043 ts_tablet_manager.cc:1397] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.125s user 0.029s sys 0.044s
I20250626 08:22:42.623718 2045 raft_consensus.cc:357] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.624364 2045 raft_consensus.cc:383] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.624624 2045 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.624583 2043 raft_consensus.cc:357] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.625324 2043 raft_consensus.cc:383] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.625579 2043 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.625988 2045 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.626358 2043 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.628044 2045 ts_tablet_manager.cc:1428] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250626 08:22:42.628922 2045 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.628971 2043 ts_tablet_manager.cc:1428] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:22:42.629287 2044 tablet_bootstrap.cc:654] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.629959 2043 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.631706 1928 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.637259 2045 tablet_bootstrap.cc:654] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.640180 2286 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:42.640270 2043 tablet_bootstrap.cc:654] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.643060 2044 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.643800 2044 ts_tablet_manager.cc:1397] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.005s
I20250626 08:22:42.647145 2044 raft_consensus.cc:357] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.648025 2105 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:42.648391 2044 raft_consensus.cc:383] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.648810 2044 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.650270 2105 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.650509 2044 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.650892 2105 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.654266 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:42.655558 1968 leader_election.cc:343] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.656996 2045 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.657478 2045 ts_tablet_manager.cc:1397] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.011s
I20250626 08:22:42.657419 2105 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.654395 2044 ts_tablet_manager.cc:1428] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20250626 08:22:42.658802 2044 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.662086 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.663563 2045 raft_consensus.cc:357] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.664245 2045 raft_consensus.cc:383] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.664495 2044 tablet_bootstrap.cc:654] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.664556 2045 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
W20250626 08:22:42.665694 1967 leader_election.cc:343] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.666206 1967 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.666013 2045 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.673035 2292 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.673714 2292 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.674181 2292 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.676290 2292 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.678835 2045 ts_tablet_manager.cc:1428] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250626 08:22:42.679868 2045 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.680538 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.681535 1847 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.681756 2279 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.682350 2279 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.683465 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.684515 2279 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.686165 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.687444 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.688740 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.688216 1847 raft_consensus.cc:2391] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:42.690910 1893 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.691902 2279 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.694489 2177 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.695013 2177 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.697865 2177 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.693749 2045 tablet_bootstrap.cc:654] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.699183 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.699507 2279 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250626 08:22:42.701404 1816 leader_election.cc:343] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.694114 2105 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.702553 2105 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.703063 2105 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.707747 2045 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.708248 2045 ts_tablet_manager.cc:1397] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20250626 08:22:42.712244 2044 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.712749 2044 ts_tablet_manager.cc:1397] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.054s user 0.026s sys 0.008s
W20250626 08:22:42.714416 1967 leader_election.cc:343] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.715333 2044 raft_consensus.cc:357] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.716050 2044 raft_consensus.cc:383] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.716326 2044 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.716960 2044 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.718942 2044 ts_tablet_manager.cc:1428] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250626 08:22:42.719944 2044 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.720007 2045 raft_consensus.cc:357] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.724766 2045 raft_consensus.cc:383] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.725078 2045 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.726225 2044 tablet_bootstrap.cc:654] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.725716 2045 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.728547 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.732851 2045 ts_tablet_manager.cc:1428] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250626 08:22:42.733829 2045 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.741392 2044 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.741479 1847 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.742013 2044 ts_tablet_manager.cc:1397] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
W20250626 08:22:42.742107 1814 leader_election.cc:343] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.742619 1814 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.743902 2177 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.744676 2044 raft_consensus.cc:357] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.745332 2044 raft_consensus.cc:383] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.745610 2044 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.746392 2044 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.748323 2044 ts_tablet_manager.cc:1428] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20250626 08:22:42.750263 2292 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:42.751951 2044 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.753937 2043 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.754159 2045 tablet_bootstrap.cc:654] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.754549 2043 ts_tablet_manager.cc:1397] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.003s
I20250626 08:22:42.757992 2044 tablet_bootstrap.cc:654] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.762163 2043 raft_consensus.cc:357] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.764138 2045 tablet_bootstrap.cc:492] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.768658 2045 ts_tablet_manager.cc:1397] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20250626 08:22:42.768199 2043 raft_consensus.cc:383] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.769070 2043 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.771202 2043 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.771586 2045 raft_consensus.cc:357] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.772222 2045 raft_consensus.cc:383] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.772523 2045 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.773118 2045 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.774365 2043 ts_tablet_manager.cc:1428] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250626 08:22:42.775220 2045 ts_tablet_manager.cc:1428] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:22:42.767455 2105 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.776135 2045 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.779572 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:42.781072 1967 leader_election.cc:343] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.781971 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.775311 2043 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.782557 1849 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.782680 2044 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.783188 2044 ts_tablet_manager.cc:1397] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20250626 08:22:42.783504 2105 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.785142 2045 tablet_bootstrap.cc:654] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.785537 2044 raft_consensus.cc:357] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.786289 2044 raft_consensus.cc:383] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.786626 2044 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.787321 2044 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.789901 2044 ts_tablet_manager.cc:1428] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:42.790185 1849 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:42.790882 2044 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.791033 2043 tablet_bootstrap.cc:654] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.791334 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.792737 2105 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.793205 2105 raft_consensus.cc:695] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:42.794065 2105 consensus_queue.cc:237] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.797268 2044 tablet_bootstrap.cc:654] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.798441 2045 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.798978 2045 ts_tablet_manager.cc:1397] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.013s
I20250626 08:22:42.801384 2045 raft_consensus.cc:357] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.802140 2045 raft_consensus.cc:383] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.802387 2045 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.802959 2045 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.805089 2045 ts_tablet_manager.cc:1428] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:42.805905 2045 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.813889 2044 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.814388 2044 ts_tablet_manager.cc:1397] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.011s
I20250626 08:22:42.816691 2044 raft_consensus.cc:357] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.817335 2044 raft_consensus.cc:383] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.817598 2044 raft_consensus.cc:738] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.817611 2045 tablet_bootstrap.cc:654] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.821097 2044 consensus_queue.cc:260] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.823331 2044 ts_tablet_manager.cc:1428] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250626 08:22:42.824148 2044 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.826593 2279 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.827240 2043 tablet_bootstrap.cc:492] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.827056 2279 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.827704 2043 ts_tablet_manager.cc:1397] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.004s
I20250626 08:22:42.829497 2044 tablet_bootstrap.cc:654] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.830296 2043 raft_consensus.cc:357] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.830957 2043 raft_consensus.cc:383] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.831214 2043 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.831821 2043 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.834627 2043 ts_tablet_manager.cc:1428] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250626 08:22:42.834735 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.835364 1849 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:42.835915 2279 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.836916 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:42.837682 1744 catalog_manager.cc:5582] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:42.838508 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.839422 1998 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:42.841395 2043 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.838840 2279 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.849341 2279 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.849742 2279 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.854100 2064 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.854559 2064 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.856194 2064 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.856462 2279 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.846071 2044 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:42.848879 2177 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.857800 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.858244 2044 ts_tablet_manager.cc:1397] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.003s
I20250626 08:22:42.858198 2177 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:42.859453 1816 leader_election.cc:343] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.860040 2177 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.860939 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.860904 2044 raft_consensus.cc:357] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.861626 2044 raft_consensus.cc:383] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.861627 1998 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:42.862066 2044 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.862908 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.865059 2064 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.865535 2064 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.865991 2064 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.866068 2177 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:42.868796 2043 tablet_bootstrap.cc:654] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.869335 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.869750 2105 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.870349 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.870220 2105 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.862668 2044 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.872169 2105 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.872872 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.873420 1998 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.873634 2045 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.873513 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.874122 2045 ts_tablet_manager.cc:1397] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.068s user 0.015s sys 0.020s
I20250626 08:22:42.874848 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.875391 1847 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:42.876768 1814 leader_election.cc:343] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.876612 2045 raft_consensus.cc:357] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.877297 2045 raft_consensus.cc:383] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.877581 2045 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
W20250626 08:22:42.877720 1814 leader_election.cc:343] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.878213 2045 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.880424 1998 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:42.880605 2045 ts_tablet_manager.cc:1428] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:42.881567 2045 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
W20250626 08:22:42.881448 1967 leader_election.cc:343] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.882005 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.882751 2238 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.883213 2238 raft_consensus.cc:695] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:42.883606 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.884023 2238 consensus_queue.cc:237] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:42.885069 1968 leader_election.cc:343] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.885555 1968 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.886511 2288 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.891108 2045 tablet_bootstrap.cc:654] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.878194 1814 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:42.891546 2044 ts_tablet_manager.cc:1428] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250626 08:22:42.870368 2279 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.892583 2177 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:42.881740 1847 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:42.894068 2064 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.900591 2045 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:42.902745 2045 ts_tablet_manager.cc:1397] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20250626 08:22:42.900617 2043 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:42.904063 2043 ts_tablet_manager.cc:1397] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.009s
I20250626 08:22:42.904796 2045 raft_consensus.cc:357] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.905471 2045 raft_consensus.cc:383] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.905733 2045 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.906358 2045 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.906739 2043 raft_consensus.cc:357] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.907073 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:42.908319 2288 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.908619 2045 ts_tablet_manager.cc:1428] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20250626 08:22:42.908485 1814 leader_election.cc:343] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.908813 2288 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.909590 2045 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:42.909338 2286 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:42.907418 2043 raft_consensus.cc:383] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:42.910787 2043 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:42.911180 2288 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.912374 2044 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:42.917181 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.917865 1847 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.918416 2064 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.918800 2044 tablet_bootstrap.cc:654] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.919302 1968 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.919482 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:42.920069 1999 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.920239 2105 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.920666 2105 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.920979 2105 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.911512 2043 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.924031 2043 ts_tablet_manager.cc:1428] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250626 08:22:42.925091 2043 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:42.929780 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.932956 2045 tablet_bootstrap.cc:654] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:42.933465 1967 leader_election.cc:343] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.934571 2043 tablet_bootstrap.cc:654] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:42.932189 1999 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:42.935858 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:42.938511 2326 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:42.939270 2326 raft_consensus.cc:695] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:42.939950 2326 consensus_queue.cc:237] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.938261 2105 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.954727 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:42.955497 1847 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.959096 2288 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.959796 2288 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.962983 2288 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:42.963403 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.964249 1849 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.965835 1968 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:42.968323 2288 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:42.968950 2288 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:42.969390 2288 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:42.968391 1745 catalog_manager.cc:5582] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:42.974498 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:42.975441 1927 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:42.964326 2064 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.983541 2064 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.987810 2064 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:42.988605 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:42.990554 1816 leader_election.cc:343] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:42.992799 2279 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:42.993793 2279 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:42.998132 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:42.999107 2105 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.000872 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:42.981767 1847 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:43.009477 1849 raft_consensus.cc:2391] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:43.008253 2279 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.043143 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.044368 2044 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.046693 1968 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.048204 2044 ts_tablet_manager.cc:1397] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.136s user 0.026s sys 0.026s
I20250626 08:22:43.048373 1893 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:22:43.051798 1967 leader_election.cc:343] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.044677 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.050148 2279 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.053169 2292 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
W20250626 08:22:43.062148 1814 leader_election.cc:343] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.062608 1814 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:43.065142 2105 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.066438 2292 raft_consensus.cc:695] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:43.067272 2177 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.077471 2292 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.088655 2045 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.089175 2045 ts_tablet_manager.cc:1397] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.006s
I20250626 08:22:43.096902 2279 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.096185 2044 raft_consensus.cc:357] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.103575 2044 raft_consensus.cc:383] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.103955 2044 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.104081 2043 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.104590 2043 ts_tablet_manager.cc:1397] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.180s user 0.009s sys 0.016s
I20250626 08:22:43.105003 2184 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.106977 2043 raft_consensus.cc:357] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.107582 2043 raft_consensus.cc:383] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.107754 2043 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.108196 2043 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.101728 2045 raft_consensus.cc:357] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.109231 2045 raft_consensus.cc:383] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.109512 2045 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.110682 2045 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.113214 2045 ts_tablet_manager.cc:1428] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250626 08:22:43.101444 2288 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.115713 2043 ts_tablet_manager.cc:1428] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250626 08:22:43.110338 2044 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.116050 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:43.116782 1849 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.118024 2043 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.119031 2045 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.130391 2184 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.130754 2288 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.130945 2184 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.132539 2044 ts_tablet_manager.cc:1428] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.084s user 0.005s sys 0.002s
I20250626 08:22:43.133484 2044 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.135720 2184 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.138257 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.139012 1848 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:43.140233 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.141801 1849 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:43.142576 2045 tablet_bootstrap.cc:654] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.139256 1745 catalog_manager.cc:5582] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:43.142982 1968 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.144301 2044 tablet_bootstrap.cc:654] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.147444 2105 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.143141 2184 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.147899 2184 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.147928 2105 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:43.145627 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.109038 1744 catalog_manager.cc:5582] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:43.148329 2184 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.148561 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.148783 2105 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.148739 1927 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.159946 1927 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
W20250626 08:22:43.161329 1967 leader_election.cc:343] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.164448 2043 tablet_bootstrap.cc:654] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.181857 2290 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.183364 2290 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.177328 2184 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.185403 2290 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.182278 2044 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.190713 2044 ts_tablet_manager.cc:1397] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20250626 08:22:43.193073 2184 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.193444 2043 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.193948 2043 ts_tablet_manager.cc:1397] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.012s
I20250626 08:22:43.195935 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.197573 2043 raft_consensus.cc:357] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.198261 2043 raft_consensus.cc:383] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.198534 2043 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.199107 2043 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.199821 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:43.200449 1847 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:43.201444 1967 leader_election.cc:343] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.201581 2043 ts_tablet_manager.cc:1428] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:43.201520 2105 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:43.202631 2043 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.204146 2184 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.209316 1847 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:43.207957 2044 raft_consensus.cc:357] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.214381 2044 raft_consensus.cc:383] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.213582 1746 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:43.214669 2044 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.215195 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:43.215713 2184 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.215437 2044 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.216425 2336 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.216176 2184 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.212695 2043 tablet_bootstrap.cc:654] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.216926 2336 raft_consensus.cc:695] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:43.217963 2336 consensus_queue.cc:237] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.219664 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.220525 1848 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:43.221810 1968 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.222762 2288 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.223248 2288 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.223686 2288 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.230249 2044 ts_tablet_manager.cc:1428] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20250626 08:22:43.231496 2044 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.232604 2288 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.235466 2045 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.235563 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:43.236037 2045 ts_tablet_manager.cc:1397] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.117s user 0.011s sys 0.012s
I20250626 08:22:43.236241 1848 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.236164 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.236863 1926 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.243522 1926 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:43.244928 1967 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.245640 2105 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.245975 2288 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.248636 2105 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:43.248850 1848 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:43.249662 2105 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:43.250387 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 68388.5us
I20250626 08:22:43.257889 2045 raft_consensus.cc:357] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.258311 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.258683 2045 raft_consensus.cc:383] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.258963 2045 raft_consensus.cc:738] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.259585 2045 consensus_queue.cc:260] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.259850 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.262107 2045 ts_tablet_manager.cc:1428] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20250626 08:22:43.259042 1849 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:43.264003 2184 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.265522 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.266443 1926 raft_consensus.cc:2374] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1. Re-sending same reply.
I20250626 08:22:43.276625 2045 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.286319 1746 catalog_manager.cc:5582] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:43.298445 2044 tablet_bootstrap.cc:654] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.300504 1893 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:43.301268 2344 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.302891 2291 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:43.307094 2043 tablet_bootstrap.cc:492] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.307590 2043 ts_tablet_manager.cc:1397] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.105s user 0.023s sys 0.032s
I20250626 08:22:43.317678 2344 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.320652 2043 raft_consensus.cc:357] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.321347 2043 raft_consensus.cc:383] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.321633 2043 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.321877 2045 tablet_bootstrap.cc:654] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.322264 2043 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.324463 2043 ts_tablet_manager.cc:1428] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20250626 08:22:43.325387 2043 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.327035 2045 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.327523 2045 ts_tablet_manager.cc:1397] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.002s
I20250626 08:22:43.331012 2043 tablet_bootstrap.cc:654] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.332516 2184 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.329885 2045 raft_consensus.cc:357] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.333549 2045 raft_consensus.cc:383] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.333869 2045 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.333412 2184 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.334465 2045 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.335489 2291 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.336272 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:43.337785 1968 leader_election.cc:343] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.339501 2044 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.339958 2044 ts_tablet_manager.cc:1397] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.011s
I20250626 08:22:43.342358 2044 raft_consensus.cc:357] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.343050 2044 raft_consensus.cc:383] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.343326 2044 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.343657 2184 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.344882 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.344520 1744 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:43.345863 2043 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.346401 2043 ts_tablet_manager.cc:1397] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.005s
W20250626 08:22:43.346686 1967 leader_election.cc:343] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.347107 1967 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:43.347896 2184 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.349431 2043 raft_consensus.cc:357] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.350150 2043 raft_consensus.cc:383] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.350512 2043 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.351434 2043 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.354804 2043 ts_tablet_manager.cc:1428] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250626 08:22:43.355970 2043 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.363009 2043 tablet_bootstrap.cc:654] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.370347 1927 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.372911 2044 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.383503 2326 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.386682 2044 ts_tablet_manager.cc:1428] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250626 08:22:43.387749 2044 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.394937 2045 ts_tablet_manager.cc:1428] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.067s user 0.005s sys 0.002s
I20250626 08:22:43.401855 2045 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.416050 2043 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.416267 2286 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.416589 2043 ts_tablet_manager.cc:1397] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.022s
I20250626 08:22:43.419127 2043 raft_consensus.cc:357] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.419819 2043 raft_consensus.cc:383] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.420109 2043 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.420706 2043 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.422436 2043 ts_tablet_manager.cc:1428] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250626 08:22:43.422796 2044 tablet_bootstrap.cc:654] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.423287 2043 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.425944 2279 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.426440 2279 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.428314 2279 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.428975 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.429217 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:43.430222 1893 leader_election.cc:343] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:22:43.431465 1893 leader_election.cc:343] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.431860 1893 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:43.433846 2043 tablet_bootstrap.cc:654] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.439208 2045 tablet_bootstrap.cc:654] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.439371 2279 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.459131 2044 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.459517 2044 ts_tablet_manager.cc:1397] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.003s
I20250626 08:22:43.462718 2044 raft_consensus.cc:357] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.463502 2044 raft_consensus.cc:383] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.463794 2044 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.464381 2044 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.466245 2045 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.466509 2044 ts_tablet_manager.cc:1428] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250626 08:22:43.466713 2045 ts_tablet_manager.cc:1397] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.008s
I20250626 08:22:43.467978 2044 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.469213 2045 raft_consensus.cc:357] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.469889 2045 raft_consensus.cc:383] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.470181 2045 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.480082 2045 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.482337 2045 ts_tablet_manager.cc:1428] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20250626 08:22:43.483242 2044 tablet_bootstrap.cc:654] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.483778 2184 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.483958 2045 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.484287 2184 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.486210 2184 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.487710 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.488010 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:22:43.489178 1967 leader_election.cc:343] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.489811 2045 tablet_bootstrap.cc:654] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:43.491238 1968 leader_election.cc:343] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.493439 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:43.497107 2043 tablet_bootstrap.cc:492] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.497562 2043 ts_tablet_manager.cc:1397] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.002s
I20250626 08:22:43.498349 2105 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:43.499972 2043 raft_consensus.cc:357] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.500654 2043 raft_consensus.cc:383] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.500908 2043 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.501504 2043 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.504521 2043 ts_tablet_manager.cc:1428] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:22:43.506227 2043 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.507772 2044 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.508214 2044 ts_tablet_manager.cc:1397] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20250626 08:22:43.511314 2184 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.515367 2043 tablet_bootstrap.cc:654] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.518007 2044 raft_consensus.cc:357] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.518721 2044 raft_consensus.cc:383] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.519029 2044 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.519886 2044 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.523069 2044 ts_tablet_manager.cc:1428] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20250626 08:22:43.523944 2044 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.527158 2045 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.527822 2045 ts_tablet_manager.cc:1397] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.018s
I20250626 08:22:43.532753 2045 raft_consensus.cc:357] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.533924 2045 raft_consensus.cc:383] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.534251 2045 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.535650 2045 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.539255 2291 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.539398 2045 ts_tablet_manager.cc:1428] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250626 08:22:43.540156 2043 tablet_bootstrap.cc:492] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.539773 2291 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.540475 2043 ts_tablet_manager.cc:1397] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.002s
I20250626 08:22:43.542356 2043 raft_consensus.cc:357] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.543159 2043 raft_consensus.cc:383] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.543401 2043 raft_consensus.cc:738] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.541703 2291 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.545493 2044 tablet_bootstrap.cc:654] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.544606 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.546309 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:43.548564 1893 leader_election.cc:343] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.548719 2000 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:43.544009 2043 consensus_queue.cc:260] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.551263 2043 ts_tablet_manager.cc:1428] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250626 08:22:43.551139 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:43.553236 2043 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.553192 2291 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.554100 2291 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.554642 2291 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:43.554792 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:43.558928 2043 tablet_bootstrap.cc:654] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.563719 2043 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.564085 2043 ts_tablet_manager.cc:1397] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250626 08:22:43.565806 2043 raft_consensus.cc:357] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.566362 2043 raft_consensus.cc:383] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.566577 2043 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.567375 2045 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.567101 2043 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.569370 2043 ts_tablet_manager.cc:1428] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250626 08:22:43.573871 2045 tablet_bootstrap.cc:654] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.583766 2043 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.589663 2105 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:43.605585 2043 tablet_bootstrap.cc:654] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.624478 2064 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.625169 2064 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.628715 2291 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.634486 2064 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.639740 2044 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:43.640306 2044 ts_tablet_manager.cc:1397] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.117s user 0.032s sys 0.051s
I20250626 08:22:43.640833 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.641503 2000 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:43.643025 1816 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.643947 2286 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.644280 2286 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.644583 2286 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.647692 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:43.649533 2336 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:22:43.650439 1893 leader_election.cc:343] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.652832 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:43.634666 2291 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.653506 2000 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.655597 2286 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.659968 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:22:43.661439 1814 leader_election.cc:343] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.642990 2044 raft_consensus.cc:357] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.662220 2044 raft_consensus.cc:383] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.662554 2044 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.663316 2044 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.664945 2286 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.665407 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:43.668462 1999 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.664499 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.666951 2044 ts_tablet_manager.cc:1428] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.026s user 0.004s sys 0.001s
W20250626 08:22:43.669934 1814 leader_election.cc:343] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.670375 2044 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:43.670743 2000 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:43.672972 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:43.674151 2279 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.674695 2279 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:43.675616 2279 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.682367 2043 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.682847 2043 ts_tablet_manager.cc:1397] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.004s
I20250626 08:22:43.685876 2043 raft_consensus.cc:357] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.686846 2043 raft_consensus.cc:383] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.687191 2043 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.687987 2043 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.691148 1999 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:43.692463 2043 ts_tablet_manager.cc:1428] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250626 08:22:43.693689 2043 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.696951 1816 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:43.698127 2286 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.698596 2286 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:43.699364 2286 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.701570 2043 tablet_bootstrap.cc:654] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.714251 2064 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.714731 2064 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.716384 2064 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.717893 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.718493 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.719190 1924 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
W20250626 08:22:43.719367 1816 leader_election.cc:343] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.721096 1814 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:43.721295 2064 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.721923 2064 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.722079 2326 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.722975 2326 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.722198 2044 tablet_bootstrap.cc:654] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.723306 2326 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.725566 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.726312 1999 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:43.727113 1816 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.727927 2177 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.728240 2064 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.728376 2177 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.728950 2177 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.729207 2045 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.729789 2045 ts_tablet_manager.cc:1397] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.163s user 0.061s sys 0.071s
I20250626 08:22:43.733024 2045 raft_consensus.cc:357] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.733719 2045 raft_consensus.cc:383] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.734004 2045 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.734598 2045 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.735175 2043 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.735606 2043 ts_tablet_manager.cc:1397] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.006s
I20250626 08:22:43.737259 2045 ts_tablet_manager.cc:1428] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250626 08:22:43.738019 2043 raft_consensus.cc:357] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.738703 2043 raft_consensus.cc:383] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.738978 2043 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.738183 2045 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.739548 2043 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.742049 2043 ts_tablet_manager.cc:1428] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20250626 08:22:43.743229 2043 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.743506 2105 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.749238 2043 tablet_bootstrap.cc:654] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.749533 2045 tablet_bootstrap.cc:654] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:43.754623 2064 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.755226 2064 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.758601 2064 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.758881 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.759703 1999 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:43.761124 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.762243 2064 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.762677 2064 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.763005 2064 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.771292 2064 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.753242 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.774668 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:43.775432 1999 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.778609 2064 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.792466 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.793392 1925 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:43.794945 1999 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:43.797679 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.798048 2326 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.784397 1924 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:43.801523 2279 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.803299 2177 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.805541 2043 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:43.808235 2279 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.811792 2064 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:43.811421 2043 ts_tablet_manager.cc:1397] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.006s
I20250626 08:22:43.814075 2279 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.815135 2177 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.815531 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.816448 1849 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:43.816764 2326 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.818344 2380 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.818168 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:43.819875 2279 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.820355 2279 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.820791 2279 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.818974 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.826390 1927 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.832940 1924 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.834488 2184 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.835114 2184 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.836634 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.837476 1925 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.839852 2064 raft_consensus.cc:695] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:43.840559 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:43.840651 2064 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.841177 1928 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.841415 2184 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.845878 2043 raft_consensus.cc:357] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.846563 2043 raft_consensus.cc:383] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.846781 2043 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.847290 2043 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.848163 2336 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:43.848748 2336 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.851025 2045 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:43.851483 2045 ts_tablet_manager.cc:1397] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.113s user 0.037s sys 0.025s
I20250626 08:22:43.860486 2043 ts_tablet_manager.cc:1428] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.048s user 0.004s sys 0.000s
I20250626 08:22:43.839483 2380 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.865877 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:43.866508 1999 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:43.845353 2279 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.870182 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:43.870990 1848 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:43.873181 2043 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:43.864035 2336 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:43.876595 1927 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:43.858487 2292 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250626 08:22:43.884301 2045 raft_consensus.cc:357] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.885010 2045 raft_consensus.cc:383] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:43.885265 2045 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:43.885850 2045 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:43.886384 2380 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.896759 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:43.899884 2045 ts_tablet_manager.cc:1428] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.048s user 0.007s sys 0.005s
I20250626 08:22:43.906926 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:43.904554 1814 leader_election.cc:343] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:43.918934 2045 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:43.922079 2105 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:43.917618 1746 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:43.926596 1968 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:43.928555 2184 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:43.928694 1925 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:43.928952 2184 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:43.929293 2184 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:43.930524 2336 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:43.951746 2279 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:43.958832 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.005975 2044 tablet_bootstrap.cc:492] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:44.009086 2044 ts_tablet_manager.cc:1397] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.339s user 0.044s sys 0.082s
I20250626 08:22:43.998631 2000 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.004695 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.007467 2291 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.018934 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:44.004724 1928 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:44.020254 1927 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
W20250626 08:22:43.995515 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67129.2us
I20250626 08:22:44.023814 2043 tablet_bootstrap.cc:654] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:44.023977 1968 leader_election.cc:343] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.023350 2291 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.030647 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.030929 2045 tablet_bootstrap.cc:654] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.030831 2291 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.007587 2184 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.015421 1814 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:44.034768 2378 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.035210 2378 raft_consensus.cc:695] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:44.036089 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:44.035993 2378 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.036584 1998 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:44.037079 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:44.037693 1925 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:44.033423 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:44.040902 1849 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:44.043682 1816 leader_election.cc:343] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.044828 2001 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:44.047096 2184 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.032778 2388 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.050093 2000 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:44.049984 2388 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.055531 1967 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:44.060285 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:44.061669 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.051890 1816 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:44.062412 2184 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250626 08:22:44.051983 2388 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.099478 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:44.100742 1921 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:44.112906 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:44.074689 1846 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:44.108992 2382 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:44.190768 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:44.142776 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 79785.5us
I20250626 08:22:44.147635 1920 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:44.131323 2388 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:44.194728 2388 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:44.195102 2388 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.195300 2044 raft_consensus.cc:357] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.196321 2044 raft_consensus.cc:383] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.196627 2044 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.205888 2044 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.196092 2382 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:44.235980 1849 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:44.225703 2382 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:44.243710 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 188271us
I20250626 08:22:44.251955 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
W20250626 08:22:44.223794 2390 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20250626 08:22:44.258075 2044 ts_tablet_manager.cc:1428] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.249s user 0.009s sys 0.000s
I20250626 08:22:44.259091 2044 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:44.267539 2386 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.212057 1814 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:44.266223 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:44.253154 2184 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.277593 1967 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:44.277145 2336 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:44.271893 2386 raft_consensus.cc:695] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:44.289179 2043 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:44.289593 2043 ts_tablet_manager.cc:1397] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.417s user 0.038s sys 0.039s
I20250626 08:22:44.281142 2380 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.292109 2399 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.285279 2336 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:44.294027 2336 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.280490 2184 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:44.282768 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 79406.9us
I20250626 08:22:44.313087 2044 tablet_bootstrap.cc:654] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.315133 2184 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.316255 2043 raft_consensus.cc:357] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.317013 2043 raft_consensus.cc:383] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.317260 2043 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.304677 2380 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:44.317880 2043 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.325076 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:44.326381 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.328581 2045 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:44.329104 2045 ts_tablet_manager.cc:1397] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.413s user 0.018s sys 0.006s
W20250626 08:22:44.329667 1968 leader_election.cc:343] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.323583 1846 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.332969 2043 ts_tablet_manager.cc:1428] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
I20250626 08:22:44.333654 2043 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:44.320453 2386 consensus_queue.cc:237] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.361688 2388 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.346220 2380 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.392971 2290 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 08:22:44.459082 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 86506.5us
W20250626 08:22:44.459830 1967 leader_election.cc:343] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.460247 1967 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:44.437361 2382 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.464551 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:22:44.466161 1968 leader_election.cc:343] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.452171 2336 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.467432 2388 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.469875 2043 tablet_bootstrap.cc:654] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.468329 2382 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.478493 2408 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.481586 2045 raft_consensus.cc:357] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.482694 2045 raft_consensus.cc:383] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.483075 2045 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.484553 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:44.484989 2045 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.470228 2336 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.497087 1745 catalog_manager.cc:5582] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:44.502396 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:44.508376 2045 ts_tablet_manager.cc:1428] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.177s user 0.010s sys 0.009s
I20250626 08:22:44.510125 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:44.510491 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:44.509403 2045 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:44.511039 1928 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.514176 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:44.514808 1923 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.472627 2380 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20250626 08:22:44.502885 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:44.534597 2401 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250626 08:22:44.537978 2044 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:44.538492 2044 ts_tablet_manager.cc:1397] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.280s user 0.010s sys 0.007s
I20250626 08:22:44.542964 2392 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:22:44.530467 1846 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.544224 2379 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250626 08:22:44.548409 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.549062 1997 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.560588 2379 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:44.562386 2044 raft_consensus.cc:357] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.563146 2044 raft_consensus.cc:383] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.563421 2044 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.564013 2044 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.570223 2044 ts_tablet_manager.cc:1428] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20250626 08:22:44.571194 2044 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:44.590839 2043 tablet_bootstrap.cc:492] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:44.591365 2043 ts_tablet_manager.cc:1397] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.258s user 0.018s sys 0.021s
I20250626 08:22:44.610026 1997 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:22:44.632097 1814 consensus_peers.cc:487] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Couldn't send request to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250626 08:22:44.632573 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 119707us
I20250626 08:22:44.637518 1816 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:44.637759 2043 raft_consensus.cc:357] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.638784 2043 raft_consensus.cc:383] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.603051 1928 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:44.639284 2043 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
W20250626 08:22:44.643388 1814 leader_election.cc:343] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.642411 2044 tablet_bootstrap.cc:654] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.643308 2043 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:44.647205 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:44.651134 1846 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:44.646410 2382 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.654193 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:44.655817 2336 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.656311 2336 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:44.656356 1923 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:44.658265 2382 raft_consensus.cc:695] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:44.661201 2045 tablet_bootstrap.cc:654] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.657282 2336 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.654167 1744 catalog_manager.cc:5582] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:44.664548 2043 ts_tablet_manager.cc:1428] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20250626 08:22:44.664237 1744 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:44.668553 2043 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:44.669157 1967 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:44.670656 2292 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:44.671134 2292 raft_consensus.cc:695] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:44.671453 2382 consensus_queue.cc:237] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.671492 1744 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:44.671898 2292 consensus_queue.cc:237] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.681391 2380 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:44.702497 2044 tablet_bootstrap.cc:492] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:44.702986 2044 ts_tablet_manager.cc:1397] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.132s user 0.016s sys 0.001s
I20250626 08:22:44.727183 2043 tablet_bootstrap.cc:654] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.728181 2044 raft_consensus.cc:357] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.729050 2044 raft_consensus.cc:383] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.729383 2044 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.730165 2044 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.733089 2064 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:44.709800 1746 catalog_manager.cc:5582] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:44.739185 2044 ts_tablet_manager.cc:1428] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250626 08:22:44.752317 2044 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:44.763455 2044 tablet_bootstrap.cc:654] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.775830 2381 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.776371 2381 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.780941 2381 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:22:44.783550 2423 log.cc:927] Time spent T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20250626 08:22:44.794018 2383 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.796310 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:44.795333 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:44.817584 1744 catalog_manager.cc:5582] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
W20250626 08:22:44.821496 1816 leader_election.cc:343] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.821076 2392 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.825232 2392 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.794569 2383 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.828928 2383 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.829658 1927 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:44.834792 2392 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.835274 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.835943 1846 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:44.836915 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:22:44.837692 2380 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:44.838708 1893 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:44.839795 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.840688 2043 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:44.841123 2043 ts_tablet_manager.cc:1397] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.173s user 0.014s sys 0.008s
I20250626 08:22:44.838053 2380 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:44.845211 2045 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:44.845824 2045 ts_tablet_manager.cc:1397] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.337s user 0.024s sys 0.031s
I20250626 08:22:44.851642 2044 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:44.852061 2044 ts_tablet_manager.cc:1397] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.112s user 0.014s sys 0.008s
I20250626 08:22:44.854212 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:44.843984 2392 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:44.855777 1996 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:44.844812 2380 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.859354 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:44.857705 2392 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:44.860977 2392 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:44.863034 1893 leader_election.cc:343] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.865296 2380 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.870747 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:44.872831 2401 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.889115 2043 raft_consensus.cc:357] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.889858 2043 raft_consensus.cc:383] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.890151 2043 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.890786 2043 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.893049 2380 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.895977 2044 raft_consensus.cc:357] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.896636 2044 raft_consensus.cc:383] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.896904 2044 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.898197 2392 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.898440 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.896179 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:44.899434 1920 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
W20250626 08:22:44.899916 1816 leader_election.cc:343] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.900177 2392 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:44.897464 2044 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.902921 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:44.903790 2044 ts_tablet_manager.cc:1428] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20250626 08:22:44.904742 2044 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:44.905377 2381 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.907064 2045 raft_consensus.cc:357] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.907774 2045 raft_consensus.cc:383] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.908037 2045 raft_consensus.cc:738] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.908957 2045 consensus_queue.cc:260] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.911978 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:44.915576 2043 ts_tablet_manager.cc:1428] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.074s user 0.006s sys 0.002s
W20250626 08:22:44.916105 1893 leader_election.cc:343] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.916595 2043 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:44.919970 2045 ts_tablet_manager.cc:1428] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20250626 08:22:44.920805 2045 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:44.915527 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:44.918340 2292 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.923023 2292 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.926440 2292 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.926872 2401 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.926892 1846 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:44.927647 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.930465 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:22:44.930907 1968 leader_election.cc:343] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.931365 1893 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:22:44.932211 1967 leader_election.cc:343] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:44.936844 2380 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250626 08:22:44.946328 2401 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.952943 2045 tablet_bootstrap.cc:654] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.953306 2043 tablet_bootstrap.cc:654] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.952265 2044 tablet_bootstrap.cc:654] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:44.945322 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:44.956142 2292 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:44.948148 1744 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:44.960120 2292 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:44.960594 2292 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.960649 1744 catalog_manager.cc:5582] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:44.965934 2043 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:44.966424 2043 ts_tablet_manager.cc:1397] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.006s
I20250626 08:22:44.969226 2043 raft_consensus.cc:357] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.969951 2043 raft_consensus.cc:383] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.970235 2043 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:44.975124 2292 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:44.975098 2043 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.974565 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:44.977864 2045 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:44.978345 2045 ts_tablet_manager.cc:1397] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.010s
I20250626 08:22:44.980736 2045 raft_consensus.cc:357] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:44.982357 2383 consensus_queue.cc:1035] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:44.984493 2045 raft_consensus.cc:383] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:44.984114 2043 ts_tablet_manager.cc:1428] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250626 08:22:44.987167 2043 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:44.991233 1997 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:44.994748 2043 tablet_bootstrap.cc:654] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.004819 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.009199 2045 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.011093 2045 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.012042 2380 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.020915 1967 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:45.021792 2045 ts_tablet_manager.cc:1428] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.043s user 0.002s sys 0.004s
I20250626 08:22:45.022856 2045 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:45.027318 2044 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.032454 2044 ts_tablet_manager.cc:1397] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.128s user 0.015s sys 0.013s
I20250626 08:22:45.032799 2388 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:45.035120 2044 raft_consensus.cc:357] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.035816 2044 raft_consensus.cc:383] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.036069 2044 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.036860 2290 consensus_queue.cc:1035] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.036665 2044 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.038602 2044 ts_tablet_manager.cc:1428] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:22:45.039469 2044 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:45.041335 2177 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.041785 2177 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.043629 2177 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.045493 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:45.046135 1997 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:45.046150 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.046854 1928 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:45.047142 1816 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:45.049306 2177 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.049634 2177 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.049902 2177 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.052678 2045 tablet_bootstrap.cc:654] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.060235 2044 tablet_bootstrap.cc:654] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.077531 2045 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.078156 2045 ts_tablet_manager.cc:1397] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.000s
I20250626 08:22:45.077878 2177 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.080770 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:45.080832 2045 raft_consensus.cc:357] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.081419 1928 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.081525 2045 raft_consensus.cc:383] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.083853 2043 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.084287 2043 ts_tablet_manager.cc:1397] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.006s
I20250626 08:22:45.085951 2177 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.086791 2043 raft_consensus.cc:357] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.087453 2043 raft_consensus.cc:383] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.087706 2043 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.081821 2045 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.090570 2045 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.091668 2043 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.092751 2045 ts_tablet_manager.cc:1428] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20250626 08:22:45.093621 2045 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:45.094996 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:45.095605 1998 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.097348 2177 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.097805 2177 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.099464 2043 ts_tablet_manager.cc:1428] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20250626 08:22:45.104992 2177 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.106943 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:45.111021 1928 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:45.111681 2043 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.112035 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.112273 1998 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:45.112363 2177 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.113619 1816 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:45.112818 1920 raft_consensus.cc:2391] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:45.114439 2380 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.114917 2380 raft_consensus.cc:695] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:45.115134 2044 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.115775 2380 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.117522 2044 ts_tablet_manager.cc:1397] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.010s
I20250626 08:22:45.119202 2043 tablet_bootstrap.cc:654] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.121675 1814 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:45.125756 2044 raft_consensus.cc:357] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.128716 2044 raft_consensus.cc:383] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.129105 2044 raft_consensus.cc:738] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.128872 2388 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:45.130950 2401 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.115787 2045 tablet_bootstrap.cc:654] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.131594 2401 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.137037 2177 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.150969 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:45.150640 2044 consensus_queue.cc:260] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:22:45.154230 1893 leader_election.cc:343] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.163878 2044 ts_tablet_manager.cc:1428] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.046s user 0.005s sys 0.001s
I20250626 08:22:45.164134 2401 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.164873 2044 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:45.166266 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:45.167189 1996 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:45.183190 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.189898 2401 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.190477 2401 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.190901 2401 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.196029 2388 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:45.199875 2177 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:45.202667 1745 catalog_manager.cc:5582] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:45.212040 2401 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.223513 2336 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:45.224311 2401 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.225703 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:45.226552 1996 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.229972 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:45.233749 2044 tablet_bootstrap.cc:654] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
W20250626 08:22:45.239935 1893 leader_election.cc:343] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.241461 2388 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.264796 2045 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.265405 2045 ts_tablet_manager.cc:1397] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.172s user 0.014s sys 0.029s
I20250626 08:22:45.269197 2382 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.271765 2045 raft_consensus.cc:357] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.272435 2045 raft_consensus.cc:383] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.272703 2045 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.272598 1996 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:45.275403 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.273447 2045 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.278025 2383 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.278568 2383 raft_consensus.cc:695] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:45.283144 2383 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.290944 2045 ts_tablet_manager.cc:1428] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250626 08:22:45.292013 2045 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:45.301681 2378 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.304512 2378 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.312089 2378 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.314210 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.314921 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:45.316627 1814 leader_election.cc:343] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:22:45.317240 1816 leader_election.cc:343] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.317672 1816 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:45.334090 2382 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:45.337247 2045 tablet_bootstrap.cc:654] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.351106 2043 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.351634 2043 ts_tablet_manager.cc:1397] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.240s user 0.046s sys 0.068s
I20250626 08:22:45.351166 2382 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.394908 2043 raft_consensus.cc:357] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.361701 2382 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.395789 2043 raft_consensus.cc:383] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.399091 2043 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
W20250626 08:22:45.404088 2452 log.cc:927] Time spent T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250626 08:22:45.403857 1744 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:45.405401 2043 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.407752 2044 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.408217 2044 ts_tablet_manager.cc:1397] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.244s user 0.012s sys 0.037s
W20250626 08:22:45.408447 2455 log.cc:927] Time spent T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.083s user 0.000s sys 0.004s
I20250626 08:22:45.396906 1928 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.442185 2044 raft_consensus.cc:357] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.443159 2044 raft_consensus.cc:383] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.443522 2044 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.444877 2044 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.450327 2382 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.450886 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:22:45.452134 1814 leader_election.cc:343] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.456072 2044 ts_tablet_manager.cc:1428] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.048s user 0.007s sys 0.001s
I20250626 08:22:45.456813 2044 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:45.460413 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:45.461050 1997 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 0.
I20250626 08:22:45.462853 1816 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:45.463755 2379 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.441260 2336 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.464191 2379 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.464607 2379 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.464373 2336 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.480125 2043 ts_tablet_manager.cc:1428] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.128s user 0.010s sys 0.007s
I20250626 08:22:45.491531 2379 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.496812 2336 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.498652 2043 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.508150 2382 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.508669 2382 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.519392 2043 tablet_bootstrap.cc:654] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.523079 2379 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.524295 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:45.525509 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.526141 1925 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:45.529558 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:45.530216 1996 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.532027 2382 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.533386 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:22:45.540414 1814 leader_election.cc:343] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.547083 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.537941 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:45.551012 1968 leader_election.cc:343] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.552099 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.553339 2457 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.553717 2457 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.554069 2457 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.556212 2045 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.556602 2045 ts_tablet_manager.cc:1397] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.265s user 0.010s sys 0.033s
I20250626 08:22:45.562687 2401 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:22:45.563277 2401 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.564999 2045 raft_consensus.cc:357] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.546358 1997 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:45.569739 2382 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250626 08:22:45.572914 1816 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:45.573366 2401 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:45.579649 2380 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.575904 2044 tablet_bootstrap.cc:654] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.570303 2045 raft_consensus.cc:383] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.581104 2045 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.579288 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:45.582823 2045 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.593807 2045 ts_tablet_manager.cc:1428] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20250626 08:22:45.594837 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:45.598821 2457 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.601612 1996 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:45.602754 1816 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:45.603489 2379 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.604005 2379 raft_consensus.cc:695] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:45.604790 2379 consensus_queue.cc:237] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.605520 2380 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:45.607486 2064 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.607784 2457 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.607982 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:22:45.609628 1968 leader_election.cc:343] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.610947 2044 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.611385 2044 ts_tablet_manager.cc:1397] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.155s user 0.010s sys 0.000s
I20250626 08:22:45.611693 1893 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:45.614042 2044 raft_consensus.cc:357] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.614679 2044 raft_consensus.cc:383] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.614917 2044 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.615489 2044 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.617671 2044 ts_tablet_manager.cc:1428] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:22:45.608188 2045 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:45.619459 2044 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:45.619637 2405 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:22:45.625042 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:45.625691 1928 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.625684 2043 tablet_bootstrap.cc:492] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.634796 2043 ts_tablet_manager.cc:1397] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.154s user 0.007s sys 0.020s
I20250626 08:22:45.637306 1928 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:45.647296 2043 raft_consensus.cc:357] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.661767 2043 raft_consensus.cc:383] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.662070 2043 raft_consensus.cc:738] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.658740 1745 catalog_manager.cc:5582] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:45.662724 2043 consensus_queue.cc:260] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.666186 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.666909 2457 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.667378 2457 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:45.668155 2457 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.658629 2045 tablet_bootstrap.cc:654] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.685110 2043 ts_tablet_manager.cc:1428] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250626 08:22:45.686223 2043 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.692225 2043 tablet_bootstrap.cc:654] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.699157 2045 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.699043 2457 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.699131 2044 tablet_bootstrap.cc:654] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.699805 2045 ts_tablet_manager.cc:1397] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.092s user 0.011s sys 0.006s
I20250626 08:22:45.699901 2457 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.702351 2045 raft_consensus.cc:357] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.703171 2045 raft_consensus.cc:383] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.703464 2045 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.704164 2045 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.706686 2045 ts_tablet_manager.cc:1428] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250626 08:22:45.707634 2045 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Bootstrap starting.
I20250626 08:22:45.708011 1746 catalog_manager.cc:5582] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:45.710892 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:22:45.713878 1968 leader_election.cc:343] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.715474 2457 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:22:45.716441 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:45.720211 2045 tablet_bootstrap.cc:654] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.725728 2043 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.726590 2043 ts_tablet_manager.cc:1397] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.041s user 0.027s sys 0.010s
I20250626 08:22:45.729415 2043 raft_consensus.cc:357] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.730288 2043 raft_consensus.cc:383] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.730623 2043 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.731268 2043 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.733404 2045 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.733773 2043 ts_tablet_manager.cc:1428] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:22:45.733876 2045 ts_tablet_manager.cc:1397] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20250626 08:22:45.735013 2043 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.737165 2045 raft_consensus.cc:357] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.737885 2045 raft_consensus.cc:383] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.738173 2045 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.738780 2045 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.740281 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.741218 2045 ts_tablet_manager.cc:1428] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250626 08:22:45.741992 2043 tablet_bootstrap.cc:654] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.742218 2045 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Bootstrap starting.
W20250626 08:22:45.744738 1967 leader_election.cc:343] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.745167 1967 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:45.746047 2457 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:45.750033 2457 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.750538 2457 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.752560 2457 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.754448 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:45.755322 1849 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:45.756726 1968 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:45.757609 2460 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.757987 2460 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.758318 2460 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.768152 2045 tablet_bootstrap.cc:654] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.768765 2460 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.776080 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:45.776928 1849 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.777405 2460 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.786499 2043 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.787106 2043 ts_tablet_manager.cc:1397] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.031s
I20250626 08:22:45.787612 1849 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:45.788983 1968 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:45.789872 2043 raft_consensus.cc:357] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.790650 2043 raft_consensus.cc:383] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.790982 2043 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.791675 2043 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.793834 2043 ts_tablet_manager.cc:1428] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250626 08:22:45.794880 2043 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.797735 2460 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.798481 2460 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.800024 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.800665 2043 tablet_bootstrap.cc:654] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.801469 2460 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.806372 2458 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:45.826136 2458 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:45.827589 2458 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.808081 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:45.809366 2044 tablet_bootstrap.cc:492] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.810168 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:45.817058 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.829154 1920 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:45.829466 2044 ts_tablet_manager.cc:1397] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.210s user 0.006s sys 0.007s
I20250626 08:22:45.819512 2401 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250626 08:22:45.831188 1968 leader_election.cc:343] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.831910 2044 raft_consensus.cc:357] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.832599 2044 raft_consensus.cc:383] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.832855 2044 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
W20250626 08:22:45.837978 1967 leader_election.cc:343] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
W20250626 08:22:45.841491 1967 leader_election.cc:343] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.843669 1967 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.841693 2045 tablet_bootstrap.cc:492] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: No bootstrap required, opened a new log
I20250626 08:22:45.842144 2044 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.844983 2045 ts_tablet_manager.cc:1397] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Time spent bootstrapping tablet: real 0.103s user 0.023s sys 0.023s
I20250626 08:22:45.847100 2457 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:45.847575 2457 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:45.847973 2457 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.855620 2045 raft_consensus.cc:357] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.856490 2045 raft_consensus.cc:383] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.856835 2045 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.857508 2045 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.863701 2045 ts_tablet_manager.cc:1428] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Time spent starting tablet: real 0.018s user 0.002s sys 0.003s
I20250626 08:22:45.864238 2457 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.865746 2043 tablet_bootstrap.cc:492] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.866281 2043 ts_tablet_manager.cc:1397] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.072s user 0.031s sys 0.021s
I20250626 08:22:45.866281 2044 ts_tablet_manager.cc:1428] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250626 08:22:45.868979 2044 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:45.870512 2457 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:45.873200 2043 raft_consensus.cc:357] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.873929 2043 raft_consensus.cc:383] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:45.874202 2043 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:45.874812 2043 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.875591 2401 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:45.879271 2043 ts_tablet_manager.cc:1428] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250626 08:22:45.880424 2043 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Bootstrap starting.
I20250626 08:22:45.881991 2044 tablet_bootstrap.cc:654] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.886722 2043 tablet_bootstrap.cc:654] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:45.886701 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:45.887449 1923 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:45.899314 1923 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:45.900218 2044 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:45.901065 2044 ts_tablet_manager.cc:1397] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
I20250626 08:22:45.902460 2043 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: No bootstrap required, opened a new log
I20250626 08:22:45.902987 2043 ts_tablet_manager.cc:1397] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.014s
I20250626 08:22:45.945628 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:45.952229 2458 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.953677 2458 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.957479 2458 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:22:45.965703 1968 leader_election.cc:343] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:45.969789 2405 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:45.981652 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:45.982810 1847 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 0.
I20250626 08:22:45.984885 1968 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:45.970872 2405 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.983353 1967 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:22:45.988549 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:45.999269 1746 catalog_manager.cc:5582] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:46.002688 2383 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:46.003729 2383 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.006794 2460 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
W20250626 08:22:46.008929 1967 leader_election.cc:343] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Illegal state: must be running to vote when last-logged opid is not known
I20250626 08:22:46.009764 2460 raft_consensus.cc:695] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:46.018682 1927 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.019343 2043 raft_consensus.cc:357] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.020318 2043 raft_consensus.cc:383] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:46.020684 2043 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Initialized, Role: FOLLOWER
I20250626 08:22:46.015750 2460 consensus_queue.cc:237] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.021647 2043 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:45.994117 2458 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:46.034279 2044 raft_consensus.cc:357] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.035666 2044 raft_consensus.cc:383] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:46.035970 2044 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:46.036347 2380 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:46.036640 2044 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.037729 2380 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.026610 2458 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:46.047015 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:46.048398 2043 ts_tablet_manager.cc:1428] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent starting tablet: real 0.145s user 0.011s sys 0.001s
I20250626 08:22:46.042124 2380 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.050184 2458 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.065693 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:46.067294 1928 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:46.070421 2290 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:46.058519 2383 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.071067 2290 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.073500 2290 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.079107 1814 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:46.080987 2378 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.056792 2405 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.092497 2379 consensus_queue.cc:1035] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:46.113010 2044 ts_tablet_manager.cc:1428] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.212s user 0.003s sys 0.010s
I20250626 08:22:46.114745 2044 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Bootstrap starting.
I20250626 08:22:46.102133 2378 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:46.128969 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:46.128858 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:46.130172 1850 raft_consensus.cc:2391] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 1: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:22:46.132122 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:46.133492 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:46.134277 1844 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:46.138769 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:46.141069 2485 consensus_queue.cc:1035] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:46.142457 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:46.143575 1998 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 0.
I20250626 08:22:46.158278 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:46.159368 2405 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 08:22:46.159837 2405 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:46.160279 2405 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.168322 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:46.169903 1893 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:22:46.185012 2486 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:46.185693 2486 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.188974 2486 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:46.189698 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:22:46.190685 1849 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:22:46.191812 2392 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.192890 2044 tablet_bootstrap.cc:654] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Neither blocks nor log segments found. Creating new log.
I20250626 08:22:46.204972 2336 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:46.187613 2458 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.211576 2458 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:46.218189 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:46.224375 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:46.225127 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:46.233160 1928 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 1: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:22:46.222965 2383 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:46.225570 1849 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.245688 1921 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.302096 1967 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:22:46.313869 1745 catalog_manager.cc:5582] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:46.301009 2460 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
W20250626 08:22:46.405124 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 103127us
W20250626 08:22:46.448518 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 175242us
I20250626 08:22:46.413198 2392 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:46.476845 2388 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
W20250626 08:22:46.519912 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 198793us
W20250626 08:22:46.532586 2466 log.cc:927] Time spent T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.199s user 0.000s sys 0.009s
W20250626 08:22:46.535063 2481 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.300s user 0.007s sys 0.017s
I20250626 08:22:46.647964 1849 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:46.637573 2405 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.650681 1921 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:22:46.679982 2388 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:46.784689 2044 tablet_bootstrap.cc:492] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: No bootstrap required, opened a new log
I20250626 08:22:46.785247 2044 ts_tablet_manager.cc:1397] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent bootstrapping tablet: real 0.671s user 0.011s sys 0.039s
W20250626 08:22:46.786618 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 451489us
W20250626 08:22:46.790071 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 113830us
I20250626 08:22:46.791296 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:46.792997 2508 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:46.793450 2508 raft_consensus.cc:695] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:22:46.793802 2484 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:46.794317 2484 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.693794 2405 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.796499 2484 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:46.798602 2044 raft_consensus.cc:357] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.799338 2044 raft_consensus.cc:383] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 08:22:46.800747 2044 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Initialized, Role: FOLLOWER
I20250626 08:22:46.802321 2508 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.801508 2044 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.852452 2506 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:46.855257 2495 consensus_queue.cc:1035] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:46.863099 2044 ts_tablet_manager.cc:1428] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20250626 08:22:46.847884 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:46.884635 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:22:46.885129 1850 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.861692 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:46.889648 1925 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:22:46.892535 1968 consensus_peers.cc:487] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:46.892699 2001 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:46.893850 2336 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:46.899464 2001 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:46.908340 1850 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:22:46.926770 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:46.965670 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:22:46.976763 2507 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 08:22:46.981321 2507 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:22:46.982273 2507 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:46.989820 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:46.990581 1998 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:22:46.992187 2503 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:22:46.992661 2503 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:22:46.992995 2503 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:22:47.018188 1795 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250626 08:22:47.036449 2503 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:47.059391 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:22:47.060098 1924 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:47.061077 2503 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:47.062264 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:22:47.062901 2002 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:22:47.073232 1924 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:22:47.075376 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:22:47.076226 2503 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:22:47.076735 2503 raft_consensus.cc:695] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:22:47.077600 2503 consensus_queue.cc:237] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:47.082209 2002 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:22:47.098865 2503 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:22:47.099470 2503 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:22:47.102399 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:22:47.102423 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:22:47.103001 2503 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:47.104349 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:22:47.105315 2503 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 08:22:47.106665 1742 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:47.137727 1744 catalog_manager.cc:5582] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 0 to 2, leader changed from <none> to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:47.153323 1745 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:22:47.163678 1850 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 08:22:47.228863 2457 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 08:22:47.243952 2508 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:47.456686 2514 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:47.471144 2392 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:47.520083 1996 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 08:22:47.521353 2503 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:47.534641 1924 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 08:22:47.536167 2503 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 08:22:48.945772 687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 08:22:48.976008 2539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:48.977384 2540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 08:22:48.977572 2542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 08:22:48.978610 687 server_base.cc:1048] running on GCE node
I20250626 08:22:48.982566 687 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 08:22:48.982824 687 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 08:22:48.983033 687 hybrid_clock.cc:648] HybridClock initialized: now 1750926168983012 us; error 0 us; skew 500 ppm
I20250626 08:22:48.983633 687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 08:22:48.986990 687 webserver.cc:469] Webserver started at http://127.0.171.196:37767/ using document root <none> and password file <none>
I20250626 08:22:48.987519 687 fs_manager.cc:362] Metadata directory not provided
I20250626 08:22:48.987797 687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 08:22:48.988118 687 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 08:22:48.989437 687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-3-root/instance:
uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
format_stamp: "Formatted at 2025-06-26 08:22:48 on dist-test-slave-660p"
I20250626 08:22:48.998965 687 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.005s
I20250626 08:22:49.002496 2547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:49.003276 687 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20250626 08:22:49.003540 687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-3-root
uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
format_stamp: "Formatted at 2025-06-26 08:22:48 on dist-test-slave-660p"
I20250626 08:22:49.003790 687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 08:22:49.036336 687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 08:22:49.037884 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 08:22:49.039479 687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 08:22:49.042001 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 08:22:49.042248 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:49.042575 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 08:22:49.042730 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 08:22:49.150166 687 rpc_server.cc:307] RPC server started. Bound to: 127.0.171.196:37733
I20250626 08:22:49.150279 2611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.171.196:37733 every 8 connection(s)
I20250626 08:22:49.160012 687 auto_leader_rebalancer-test.cc:417] add a tserver: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:49.170222 2612 heartbeater.cc:344] Connected to a master server at 127.0.171.254:42921
I20250626 08:22:49.170693 2612 heartbeater.cc:461] Registering TS with master...
I20250626 08:22:49.171473 2612 heartbeater.cc:507] Master 127.0.171.254:42921 requested a full tablet report, sending...
I20250626 08:22:49.173203 1745 ts_manager.cc:194] Registered new tserver with Master: be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:22:49.174636 1745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46332
I20250626 08:22:50.177161 2612 heartbeater.cc:499] Master 127.0.171.254:42921 was elected leader, sending a full tablet report...
I20250626 08:22:50.231006 1850 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.237376 2002 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.241984 1920 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.244619 2618 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:50.245867 2513 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.262046 2513 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.263926 1924 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.267027 2002 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:50.269634 1812 consensus_peers.cc:487] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b498813b859f40cd95ed6adf6a2d8918. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:50.281628 1745 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:50.309777 1998 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.316939 1924 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.318315 2631 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.317643 1850 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:50.333472 1965 consensus_peers.cc:487] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46151fddbbf14be68feb086aa1fee4d9. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:50.339469 2504 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.341068 2616 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.342238 1924 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.351737 1850 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.357900 1846 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.373335 1742 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:50.400745 1812 consensus_peers.cc:487] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7846e33f4b874a229eb7250f99a6db61. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:50.401607 1921 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.402107 1998 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.403412 2622 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.408782 2622 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.416603 2513 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.419018 1996 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.434496 1924 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.453527 1742 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.471184 1924 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.482074 1996 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.485204 1846 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:50.485970 2538 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:50.487253 2573 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:50.498414 2573 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:50.500375 1890 consensus_peers.cc:487] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75222ddacb444f7ba27de92b1571ada8. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:50.500430 1846 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.505517 1996 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:50.520463 1742 catalog_manager.cc:5582] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:50.745404 2651 ts_tablet_manager.cc:927] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:50.756008 2651 tablet_copy_client.cc:323] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:50.773794 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet b498813b859f40cd95ed6adf6a2d8918 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:50.774473 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet b498813b859f40cd95ed6adf6a2d8918 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-b498813b859f40cd95ed6adf6a2d8918
I20250626 08:22:50.781567 1861 tablet_copy_source_session.cc:215] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:50.786026 2651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b498813b859f40cd95ed6adf6a2d8918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:50.800976 2651 tablet_copy_client.cc:806] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:50.801620 2651 tablet_copy_client.cc:670] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:50.805614 2651 tablet_copy_client.cc:538] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:50.814254 2651 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:50.833721 2651 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:50.834602 2651 tablet_bootstrap.cc:492] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:50.835258 2651 ts_tablet_manager.cc:1397] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20250626 08:22:50.837973 2651 raft_consensus.cc:357] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.838799 2651 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:50.839460 2651 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:50.842486 2651 ts_tablet_manager.cc:1428] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250626 08:22:50.847743 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-b498813b859f40cd95ed6adf6a2d8918 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:50.848234 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-b498813b859f40cd95ed6adf6a2d8918 on tablet b498813b859f40cd95ed6adf6a2d8918 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:50.893416 2651 ts_tablet_manager.cc:927] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:50.896049 2651 tablet_copy_client.cc:323] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:50.897433 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 7846e33f4b874a229eb7250f99a6db61 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:50.898020 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 7846e33f4b874a229eb7250f99a6db61 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-7846e33f4b874a229eb7250f99a6db61
I20250626 08:22:50.898947 2658 ts_tablet_manager.cc:927] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:50.900688 2658 tablet_copy_client.cc:323] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:22:50.904501 1861 tablet_copy_source_session.cc:215] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:50.909615 2661 ts_tablet_manager.cc:927] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:50.921166 2651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7846e33f4b874a229eb7250f99a6db61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:50.921783 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 46151fddbbf14be68feb086aa1fee4d9 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:22:50.922417 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 46151fddbbf14be68feb086aa1fee4d9 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-46151fddbbf14be68feb086aa1fee4d9
I20250626 08:22:50.930716 2012 tablet_copy_source_session.cc:215] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:50.937095 2658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46151fddbbf14be68feb086aa1fee4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:50.948725 2661 tablet_copy_client.cc:323] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:22:50.957327 2658 tablet_copy_client.cc:806] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:50.957998 2658 tablet_copy_client.cc:670] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:50.961134 2658 tablet_copy_client.cc:538] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:50.962884 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 75222ddacb444f7ba27de92b1571ada8 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:22:50.963469 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 75222ddacb444f7ba27de92b1571ada8 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-75222ddacb444f7ba27de92b1571ada8
I20250626 08:22:50.979060 2651 tablet_copy_client.cc:806] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:50.985175 1938 tablet_copy_source_session.cc:215] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:50.985822 2651 tablet_copy_client.cc:670] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:50.990386 2661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75222ddacb444f7ba27de92b1571ada8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:51.005889 2651 tablet_copy_client.cc:538] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:51.006675 2658 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:51.017858 2651 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:51.023443 2661 tablet_copy_client.cc:806] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:51.024073 2661 tablet_copy_client.cc:670] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:51.029579 2661 tablet_copy_client.cc:538] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:51.046141 2661 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:51.061059 2658 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:51.061894 2658 tablet_bootstrap.cc:492] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:51.062562 2658 ts_tablet_manager.cc:1397] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.013s
I20250626 08:22:51.065219 2651 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:51.066215 2651 tablet_bootstrap.cc:492] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:51.066926 2651 ts_tablet_manager.cc:1397] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.003s
I20250626 08:22:51.066623 2658 raft_consensus.cc:357] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.067502 2658 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:51.068208 2658 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.070719 2658 ts_tablet_manager.cc:1428] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250626 08:22:51.072978 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-46151fddbbf14be68feb086aa1fee4d9 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:22:51.073405 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-46151fddbbf14be68feb086aa1fee4d9 on tablet 46151fddbbf14be68feb086aa1fee4d9 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:51.075760 2651 raft_consensus.cc:357] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.083545 2651 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:51.084216 2651 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.095546 2651 ts_tablet_manager.cc:1428] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250626 08:22:51.098215 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-7846e33f4b874a229eb7250f99a6db61 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:51.098662 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-7846e33f4b874a229eb7250f99a6db61 on tablet 7846e33f4b874a229eb7250f99a6db61 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:51.107410 2661 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:51.108170 2661 tablet_bootstrap.cc:492] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:51.108774 2661 ts_tablet_manager.cc:1397] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.063s user 0.020s sys 0.029s
I20250626 08:22:51.111133 2661 raft_consensus.cc:357] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.111831 2661 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:51.112272 2661 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:51.115067 2661 ts_tablet_manager.cc:1428] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.006s user 0.003s sys 0.006s
I20250626 08:22:51.117378 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-75222ddacb444f7ba27de92b1571ada8 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:22:51.117795 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-75222ddacb444f7ba27de92b1571ada8 on tablet 75222ddacb444f7ba27de92b1571ada8 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:51.137055 2587 raft_consensus.cc:1215] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:51.297271 2587 raft_consensus.cc:1215] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:51.322731 2587 raft_consensus.cc:1215] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:51.416342 2587 raft_consensus.cc:1215] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:51.718643 2664 raft_consensus.cc:1062] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:51.721055 2664 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:51.727279 2587 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.727955 2002 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.728916 2664 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.729614 1924 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.730998 2664 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.733464 2622 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.744521 2664 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.746168 2587 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.756204 1998 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.759441 2616 raft_consensus.cc:1062] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:51.761744 2616 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:51.765079 1745 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.767402 1920 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.782837 2587 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.782979 1850 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.784371 2639 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.785269 2616 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.790043 1920 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.794039 2639 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.805789 2631 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.807158 1846 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.812812 1921 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.827251 1850 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:51.838876 1920 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:51.839279 2587 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:51.842085 2622 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:51.844864 2643 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:51.855268 2622 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.860345 2586 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.863276 2587 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.866084 2650 raft_consensus.cc:1062] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:51.868468 2650 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:51.872390 1921 raft_consensus.cc:2953] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.876840 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b498813b859f40cd95ed6adf6a2d8918 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:51.861574 1742 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.894946 2586 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.897845 2647 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:51.899304 1998 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.900882 2650 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.904980 1847 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:51.906948 2647 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:51.906605 1745 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.911872 1996 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:51.946584 2587 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:51.946810 1921 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:51.948196 2631 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:51.950270 2631 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:51.959494 2616 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.967156 1920 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.968448 2586 raft_consensus.cc:2953] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.970238 2538 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.976996 1996 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.979872 1850 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.983916 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet b498813b859f40cd95ed6adf6a2d8918 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:22:51.983259 2586 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:51.994892 1745 catalog_manager.cc:5582] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:52.001820 2684 tablet_replica.cc:331] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:22:52.006141 2684 raft_consensus.cc:2241] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:52.006911 2684 raft_consensus.cc:2270] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:52.007347 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 46151fddbbf14be68feb086aa1fee4d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:51.983950 1744 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.014840 2684 ts_tablet_manager.cc:1905] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:52.027519 2664 raft_consensus.cc:1062] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:52.030925 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 46151fddbbf14be68feb086aa1fee4d9 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:22:52.039011 2664 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:52.049520 2684 ts_tablet_manager.cc:1918] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:52.050206 2684 log.cc:1199] T b498813b859f40cd95ed6adf6a2d8918 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/b498813b859f40cd95ed6adf6a2d8918
I20250626 08:22:52.056416 1920 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:52.057200 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet b498813b859f40cd95ed6adf6a2d8918 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:52.061652 2686 tablet_replica.cc:331] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:22:52.063664 2686 raft_consensus.cc:2241] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:52.064327 1847 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:52.064409 2586 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:52.064339 2686 raft_consensus.cc:2270] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:52.066548 2650 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:52.068364 2650 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:52.074023 1920 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:52.075387 2643 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:52.077008 1996 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:52.077028 2585 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:52.080199 2538 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.081961 2622 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:52.083092 2618 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:52.083215 2585 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.098682 1850 raft_consensus.cc:2953] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.100247 2686 ts_tablet_manager.cc:1905] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:52.113317 2622 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.116874 2686 ts_tablet_manager.cc:1918] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:52.117331 2686 log.cc:1199] T 46151fddbbf14be68feb086aa1fee4d9 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/46151fddbbf14be68feb086aa1fee4d9
I20250626 08:22:52.119776 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 46151fddbbf14be68feb086aa1fee4d9 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:52.128943 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 75222ddacb444f7ba27de92b1571ada8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:52.130378 1998 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.130265 1919 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.145231 1743 catalog_manager.cc:5582] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.120096 2586 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.170321 1744 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.171720 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 75222ddacb444f7ba27de92b1571ada8 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:22:52.172607 2684 tablet_replica.cc:331] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:22:52.173549 2684 raft_consensus.cc:2241] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:52.174335 2684 raft_consensus.cc:2270] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:52.178934 2684 ts_tablet_manager.cc:1905] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:52.195161 1847 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:52.197160 2684 ts_tablet_manager.cc:1918] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:52.197815 2684 log.cc:1199] T 75222ddacb444f7ba27de92b1571ada8 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/75222ddacb444f7ba27de92b1571ada8
I20250626 08:22:52.200400 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 75222ddacb444f7ba27de92b1571ada8 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:52.203454 1996 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:52.204653 2664 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:52.205215 2586 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:52.206923 2664 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:52.215112 2622 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.216193 1998 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.217105 2586 raft_consensus.cc:2953] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:52.239547 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7846e33f4b874a229eb7250f99a6db61 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:52.239472 1745 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:52.251737 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 7846e33f4b874a229eb7250f99a6db61 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:22:52.259516 2691 tablet_replica.cc:331] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:22:52.260496 2691 raft_consensus.cc:2241] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:52.261103 2691 raft_consensus.cc:2270] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:52.264076 2691 ts_tablet_manager.cc:1905] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:52.277573 2691 ts_tablet_manager.cc:1918] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:52.277878 2691 log.cc:1199] T 7846e33f4b874a229eb7250f99a6db61 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/7846e33f4b874a229eb7250f99a6db61
I20250626 08:22:52.279093 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 7846e33f4b874a229eb7250f99a6db61 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:53.306696 1850 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.312515 1998 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:53.313889 2618 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.314853 1919 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:53.315991 1812 consensus_peers.cc:487] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 736124bc239f4257b44dfc38e3acc17f. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:53.316162 2618 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.321981 2622 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.323484 2001 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.324995 1919 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.334354 1920 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.344551 1743 catalog_manager.cc:5582] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.350273 2001 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:53.351624 1850 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:53.351771 1890 consensus_peers.cc:487] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cca900160913464799d0cfa9d2866779. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:53.352737 2538 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.353538 2685 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:53.362507 2685 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.363451 2001 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.375128 1850 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.382426 1746 catalog_manager.cc:5582] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.396776 1850 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.405881 2001 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:53.406172 1812 consensus_peers.cc:487] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9996474aab3b4a5ca0e10b9f200b2abc. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:53.407471 1920 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:53.407202 2618 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.408921 2622 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.418367 2622 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.434386 1998 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.434666 1920 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.435658 1746 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:53.438545 2001 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.452096 1920 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:53.452417 1965 consensus_peers.cc:487] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f29b6256843489397f6e58f34751764. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:53.454376 1846 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:53.458053 2616 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:53.459993 2616 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:53.485075 2616 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.495371 1919 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.498822 1846 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.513698 1746 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:53.698117 2714 ts_tablet_manager.cc:927] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:53.699729 2714 tablet_copy_client.cc:323] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:53.701107 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 736124bc239f4257b44dfc38e3acc17f from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:53.701663 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 736124bc239f4257b44dfc38e3acc17f from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-736124bc239f4257b44dfc38e3acc17f
I20250626 08:22:53.708199 1861 tablet_copy_source_session.cc:215] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:53.711243 2714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736124bc239f4257b44dfc38e3acc17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:53.725834 2714 tablet_copy_client.cc:806] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:53.726428 2714 tablet_copy_client.cc:670] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:53.729647 2714 tablet_copy_client.cc:538] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:53.738284 2714 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:53.758405 2714 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:53.759152 2714 tablet_bootstrap.cc:492] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:53.759727 2714 ts_tablet_manager.cc:1397] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20250626 08:22:53.762485 2714 raft_consensus.cc:357] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.763298 2714 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:53.763940 2714 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.767594 2714 ts_tablet_manager.cc:1428] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250626 08:22:53.768961 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-736124bc239f4257b44dfc38e3acc17f received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:53.769321 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-736124bc239f4257b44dfc38e3acc17f on tablet 736124bc239f4257b44dfc38e3acc17f with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:53.790764 2714 ts_tablet_manager.cc:927] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:53.792316 2714 tablet_copy_client.cc:323] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:22:53.793745 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet cca900160913464799d0cfa9d2866779 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:22:53.794301 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet cca900160913464799d0cfa9d2866779 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-cca900160913464799d0cfa9d2866779
I20250626 08:22:53.800030 1938 tablet_copy_source_session.cc:215] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:53.802812 2714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cca900160913464799d0cfa9d2866779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:53.819914 2714 tablet_copy_client.cc:806] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:53.820500 2714 tablet_copy_client.cc:670] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:53.824787 2714 tablet_copy_client.cc:538] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:53.834600 2714 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:53.851269 2714 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:53.852104 2714 tablet_bootstrap.cc:492] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:53.852622 2714 ts_tablet_manager.cc:1397] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250626 08:22:53.855283 2714 raft_consensus.cc:357] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.857673 2714 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:53.858347 2714 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.860989 2714 ts_tablet_manager.cc:1428] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250626 08:22:53.863109 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cca900160913464799d0cfa9d2866779 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:22:53.863471 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cca900160913464799d0cfa9d2866779 on tablet cca900160913464799d0cfa9d2866779 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:53.932569 2714 ts_tablet_manager.cc:927] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:53.934974 2714 tablet_copy_client.cc:323] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:22:53.936595 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 4f29b6256843489397f6e58f34751764 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:22:53.937140 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 4f29b6256843489397f6e58f34751764 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-4f29b6256843489397f6e58f34751764
I20250626 08:22:53.943442 2012 tablet_copy_source_session.cc:215] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:53.946507 2714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f29b6256843489397f6e58f34751764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:53.962059 2714 tablet_copy_client.cc:806] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:53.962554 2714 tablet_copy_client.cc:670] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:53.967108 2714 tablet_copy_client.cc:538] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:53.974741 2714 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:53.991034 2714 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:53.992007 2714 tablet_bootstrap.cc:492] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:53.992617 2714 ts_tablet_manager.cc:1397] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.001s
I20250626 08:22:53.994603 2714 raft_consensus.cc:357] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.995337 2714 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:53.995896 2714 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:53.998036 2714 ts_tablet_manager.cc:1428] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250626 08:22:53.999425 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-4f29b6256843489397f6e58f34751764 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:22:53.999832 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-4f29b6256843489397f6e58f34751764 on tablet 4f29b6256843489397f6e58f34751764 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:54.018229 2714 ts_tablet_manager.cc:927] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:54.024456 2714 tablet_copy_client.cc:323] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:54.025965 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 9996474aab3b4a5ca0e10b9f200b2abc from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:54.026404 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 9996474aab3b4a5ca0e10b9f200b2abc from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-9996474aab3b4a5ca0e10b9f200b2abc
I20250626 08:22:54.033020 1861 tablet_copy_source_session.cc:215] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:54.036057 2714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9996474aab3b4a5ca0e10b9f200b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:54.050009 2714 tablet_copy_client.cc:806] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:54.050527 2714 tablet_copy_client.cc:670] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:54.053959 2714 tablet_copy_client.cc:538] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:54.059868 2714 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:54.075006 2714 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:54.075840 2714 tablet_bootstrap.cc:492] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:54.076342 2714 ts_tablet_manager.cc:1397] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250626 08:22:54.078318 2714 raft_consensus.cc:357] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:54.079017 2714 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:54.079514 2714 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:54.081542 2714 ts_tablet_manager.cc:1428] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250626 08:22:54.083348 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-9996474aab3b4a5ca0e10b9f200b2abc received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:54.083760 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-9996474aab3b4a5ca0e10b9f200b2abc on tablet 9996474aab3b4a5ca0e10b9f200b2abc with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:54.210126 2586 raft_consensus.cc:1215] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:54.341337 2586 raft_consensus.cc:1215] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:54.380234 2586 raft_consensus.cc:1215] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:54.636195 2586 raft_consensus.cc:1215] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:54.731782 2622 raft_consensus.cc:1062] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:54.733904 2622 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:54.739030 2586 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.739861 1998 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.740424 1919 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.741580 2622 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:54.742429 2724 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:54.743422 2692 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:54.762276 2622 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.768407 1920 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.770366 2586 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.775004 2001 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.781636 2703 raft_consensus.cc:1062] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:54.784138 2703 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:54.783284 1743 catalog_manager.cc:5582] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:54.804989 2727 raft_consensus.cc:1062] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:54.812580 2727 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:54.832342 2001 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.834317 1924 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.838568 2704 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:54.840016 2586 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.836913 2726 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250626 08:22:54.842000 2704 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:54.842147 2585 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.847697 1850 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.844163 2726 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250626 08:22:54.851397 2704 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250626 08:22:54.883775 2704 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.889560 1847 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:54.892364 1998 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.894075 2713 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:54.903012 2585 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.906148 2586 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.903640 2713 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.919456 1850 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.953024 1924 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.953763 1850 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:54.972177 1746 catalog_manager.cc:5582] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.973575 2586 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:54.976471 1998 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:54.977797 2729 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:54.978758 2622 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:54.973809 1746 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.987736 2692 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.988708 2001 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.994509 2587 raft_consensus.cc:2953] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:54.998041 1846 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.002135 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 736124bc239f4257b44dfc38e3acc17f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:55.012315 1919 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:55.019106 2586 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.020748 2737 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:55.022118 1998 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.029263 2538 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.030057 2737 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:55.033789 1744 catalog_manager.cc:5582] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:55.035956 2001 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:55.037358 1998 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.043568 2585 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.044518 1919 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.036055 2586 raft_consensus.cc:2953] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.047336 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cca900160913464799d0cfa9d2866779 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:55.058585 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 736124bc239f4257b44dfc38e3acc17f with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:22:55.057279 2713 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250626 08:22:55.060657 2727 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:55.056653 1745 catalog_manager.cc:5582] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:55.073312 2727 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.075644 1919 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.078380 2746 tablet_replica.cc:331] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:22:55.083473 2746 raft_consensus.cc:2241] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:55.084112 2746 raft_consensus.cc:2270] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:55.089058 2746 ts_tablet_manager.cc:1905] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:55.091145 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4f29b6256843489397f6e58f34751764 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:55.107900 2586 raft_consensus.cc:2953] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.126626 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet cca900160913464799d0cfa9d2866779 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:22:55.131206 1743 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:55.145987 2749 tablet_replica.cc:331] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:22:55.147209 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f29b6256843489397f6e58f34751764 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:22:55.148659 2749 raft_consensus.cc:2241] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:55.149396 2749 raft_consensus.cc:2270] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:55.153311 2749 ts_tablet_manager.cc:1905] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:55.170353 2749 ts_tablet_manager.cc:1918] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:55.170841 2749 log.cc:1199] T cca900160913464799d0cfa9d2866779 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/cca900160913464799d0cfa9d2866779
I20250626 08:22:55.172583 2749 tablet_replica.cc:331] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:22:55.173425 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet cca900160913464799d0cfa9d2866779 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:55.173646 2749 raft_consensus.cc:2241] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:55.174871 2749 raft_consensus.cc:2270] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:55.178174 2749 ts_tablet_manager.cc:1905] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:55.204051 2749 ts_tablet_manager.cc:1918] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:55.204543 2749 log.cc:1199] T 4f29b6256843489397f6e58f34751764 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/4f29b6256843489397f6e58f34751764
I20250626 08:22:55.205291 2746 ts_tablet_manager.cc:1918] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:55.205886 2746 log.cc:1199] T 736124bc239f4257b44dfc38e3acc17f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/736124bc239f4257b44dfc38e3acc17f
I20250626 08:22:55.206019 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 4f29b6256843489397f6e58f34751764 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:55.209410 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 736124bc239f4257b44dfc38e3acc17f (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:55.215910 2622 raft_consensus.cc:1062] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:55.218019 2622 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:55.231386 2001 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:55.232643 2586 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:55.234536 2724 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:55.238034 2729 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:55.245738 1919 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:55.247435 2729 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:55.253476 2692 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.261678 1996 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.263994 1919 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.266691 2586 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.274904 1743 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:55.299126 1848 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:55.311275 2586 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.311736 1996 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:55.312826 2729 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:55.313771 2622 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:55.321437 2724 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.323263 2586 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.326498 2001 raft_consensus.cc:2953] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.335947 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9996474aab3b4a5ca0e10b9f200b2abc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:55.338284 1746 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:55.349464 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 9996474aab3b4a5ca0e10b9f200b2abc with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:22:55.350342 2746 tablet_replica.cc:331] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:22:55.351366 2746 raft_consensus.cc:2241] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:55.351974 2746 raft_consensus.cc:2270] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:55.354573 2746 ts_tablet_manager.cc:1905] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:55.367067 2746 ts_tablet_manager.cc:1918] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:55.367492 2746 log.cc:1199] T 9996474aab3b4a5ca0e10b9f200b2abc P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/9996474aab3b4a5ca0e10b9f200b2abc
I20250626 08:22:55.369060 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 9996474aab3b4a5ca0e10b9f200b2abc (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:56.420900 1848 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.426815 2001 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:56.427353 1812 consensus_peers.cc:487] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9b805acc8ae4c3abfe6d6ee40fbf456. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:56.428251 1919 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.428339 2758 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:56.429795 2757 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:56.436735 2758 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.439256 2001 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.440161 1919 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.454655 1848 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.463987 1745 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:56.467443 1812 consensus_peers.cc:487] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8353d62d68914958b13bb87541c357f8. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:56.467617 1996 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.469614 2757 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:56.493161 1919 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.495025 2758 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.496965 2724 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:56.499292 2001 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.520669 1920 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.539117 1743 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.549165 1919 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.553977 2001 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.555573 2703 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250626 08:22:56.556907 1890 consensus_peers.cc:487] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4be9d96212a6463eb1aa71d0b3d300a6. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:56.558985 1847 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.560379 2737 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:56.564186 2703 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.567812 2001 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.574815 1847 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.583066 1920 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.587894 1743 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 08:22:56.595135 1890 consensus_peers.cc:487] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9122458a1eae4f07b1a7f4ff63d6be19. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:56.597672 2001 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.598775 1848 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:56.599103 2538 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:56.602372 2538 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:56.610579 2737 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.616040 2001 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.616835 1847 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:56.632546 1743 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:22:56.862613 2776 ts_tablet_manager.cc:927] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:56.865569 2776 tablet_copy_client.cc:323] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:56.867468 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:56.868034 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet f9b805acc8ae4c3abfe6d6ee40fbf456 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-f9b805acc8ae4c3abfe6d6ee40fbf456
I20250626 08:22:56.875242 1861 tablet_copy_source_session.cc:215] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:56.878327 2776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b805acc8ae4c3abfe6d6ee40fbf456. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:56.893397 2776 tablet_copy_client.cc:806] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:56.893968 2776 tablet_copy_client.cc:670] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:56.898471 2776 tablet_copy_client.cc:538] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:56.908141 2776 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:56.915575 2778 ts_tablet_manager.cc:927] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:22:56.926960 2778 tablet_copy_client.cc:323] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:22:56.928885 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 8353d62d68914958b13bb87541c357f8 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:22:56.929421 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 8353d62d68914958b13bb87541c357f8 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-8353d62d68914958b13bb87541c357f8
I20250626 08:22:56.938309 1861 tablet_copy_source_session.cc:215] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:56.938602 2776 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:56.939644 2776 tablet_bootstrap.cc:492] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:56.940372 2776 ts_tablet_manager.cc:1397] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20250626 08:22:56.943656 2776 raft_consensus.cc:357] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.944579 2776 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:56.945257 2776 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:56.953285 2778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8353d62d68914958b13bb87541c357f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:56.955225 2776 ts_tablet_manager.cc:1428] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20250626 08:22:56.970407 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-f9b805acc8ae4c3abfe6d6ee40fbf456 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:56.970954 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-f9b805acc8ae4c3abfe6d6ee40fbf456 on tablet f9b805acc8ae4c3abfe6d6ee40fbf456 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:56.978649 2778 tablet_copy_client.cc:806] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:56.979239 2778 tablet_copy_client.cc:670] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:56.982611 2778 tablet_copy_client.cc:538] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:56.991165 2778 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:56.999038 2776 ts_tablet_manager.cc:927] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:57.010509 2776 tablet_copy_client.cc:323] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:22:57.014472 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 9122458a1eae4f07b1a7f4ff63d6be19 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:22:57.014954 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 9122458a1eae4f07b1a7f4ff63d6be19 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-9122458a1eae4f07b1a7f4ff63d6be19
I20250626 08:22:57.027324 1938 tablet_copy_source_session.cc:215] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:57.030757 2776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9122458a1eae4f07b1a7f4ff63d6be19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:57.033428 2778 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:57.034452 2778 tablet_bootstrap.cc:492] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:57.035280 2778 ts_tablet_manager.cc:1397] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.044s user 0.024s sys 0.011s
I20250626 08:22:57.038123 2778 raft_consensus.cc:357] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.039108 2778 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:57.039772 2778 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.042402 2778 ts_tablet_manager.cc:1428] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250626 08:22:57.049260 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-8353d62d68914958b13bb87541c357f8 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:22:57.049721 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-8353d62d68914958b13bb87541c357f8 on tablet 8353d62d68914958b13bb87541c357f8 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:57.056088 2776 tablet_copy_client.cc:806] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:57.056752 2776 tablet_copy_client.cc:670] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:57.060514 2776 tablet_copy_client.cc:538] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:57.069687 2776 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:57.092691 2776 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:57.093605 2776 tablet_bootstrap.cc:492] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:57.094273 2776 ts_tablet_manager.cc:1397] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.012s
I20250626 08:22:57.096675 2776 raft_consensus.cc:357] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.097445 2776 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:57.098098 2776 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.102840 2776 ts_tablet_manager.cc:1428] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250626 08:22:57.104430 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-9122458a1eae4f07b1a7f4ff63d6be19 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:22:57.105247 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-9122458a1eae4f07b1a7f4ff63d6be19 on tablet 9122458a1eae4f07b1a7f4ff63d6be19 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:57.178655 2776 ts_tablet_manager.cc:927] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:22:57.180239 2776 tablet_copy_client.cc:323] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:22:57.181753 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 4be9d96212a6463eb1aa71d0b3d300a6 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:22:57.182217 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 4be9d96212a6463eb1aa71d0b3d300a6 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-4be9d96212a6463eb1aa71d0b3d300a6
I20250626 08:22:57.188901 1938 tablet_copy_source_session.cc:215] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:22:57.194878 2776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be9d96212a6463eb1aa71d0b3d300a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:22:57.209558 2776 tablet_copy_client.cc:806] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:22:57.210130 2776 tablet_copy_client.cc:670] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:22:57.213505 2776 tablet_copy_client.cc:538] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:22:57.220017 2776 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:22:57.239825 2776 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:22:57.240674 2776 tablet_bootstrap.cc:492] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:22:57.241500 2776 ts_tablet_manager.cc:1397] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.001s
I20250626 08:22:57.243589 2776 raft_consensus.cc:357] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.244158 2776 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:22:57.244711 2776 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:57.247012 2776 ts_tablet_manager.cc:1428] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250626 08:22:57.248521 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-4be9d96212a6463eb1aa71d0b3d300a6 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:22:57.248953 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-4be9d96212a6463eb1aa71d0b3d300a6 on tablet 4be9d96212a6463eb1aa71d0b3d300a6 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:22:57.318030 2586 raft_consensus.cc:1215] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:57.449715 2586 raft_consensus.cc:1215] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:57.589831 2586 raft_consensus.cc:1215] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:57.709623 2758 raft_consensus.cc:1062] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:57.712024 2758 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:57.720093 2586 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.721101 1998 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.722708 1928 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.724002 2758 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:57.724889 2757 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:57.730432 2768 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:57.747634 2768 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.755607 1998 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.759023 2585 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.770330 1928 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.773499 1746 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:57.800263 2585 raft_consensus.cc:1215] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:22:57.805335 1848 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:57.811585 2585 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:57.814129 2757 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:57.815937 2001 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:57.818387 2757 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:57.820436 2792 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.827895 2585 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.833475 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8353d62d68914958b13bb87541c357f8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:57.841197 2001 raft_consensus.cc:2953] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.856742 1743 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:57.883249 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 8353d62d68914958b13bb87541c357f8 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:22:57.888010 2768 raft_consensus.cc:1062] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:57.890846 2768 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:57.898912 2001 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.899933 2585 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.900362 2792 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:57.901201 1928 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:57.901356 2758 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:57.889544 2800 tablet_replica.cc:331] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:22:57.903573 2800 raft_consensus.cc:2241] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:57.904531 2800 raft_consensus.cc:2270] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:57.907663 2800 ts_tablet_manager.cc:1905] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:57.907661 2768 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:57.930928 2792 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.942215 2800 ts_tablet_manager.cc:1918] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:57.943650 2800 log.cc:1199] T 8353d62d68914958b13bb87541c357f8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/8353d62d68914958b13bb87541c357f8
I20250626 08:22:57.944830 2585 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.946007 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 8353d62d68914958b13bb87541c357f8 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:57.956846 2001 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.959923 1928 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.968430 1744 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:57.990218 1848 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:57.995831 2585 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:57.996853 1919 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:57.997116 2758 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:57.998216 2792 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:58.005995 2724 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.015033 1920 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.018499 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:58.018481 2585 raft_consensus.cc:2953] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.029162 1745 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:58.041481 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet f9b805acc8ae4c3abfe6d6ee40fbf456 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:22:58.043582 2805 tablet_replica.cc:331] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:22:58.044811 2805 raft_consensus.cc:2241] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:58.045598 2805 raft_consensus.cc:2270] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:58.048879 2805 ts_tablet_manager.cc:1905] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:58.064972 2805 ts_tablet_manager.cc:1918] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:58.065356 2805 log.cc:1199] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/f9b805acc8ae4c3abfe6d6ee40fbf456
I20250626 08:22:58.066969 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet f9b805acc8ae4c3abfe6d6ee40fbf456 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:58.104601 2787 raft_consensus.cc:1062] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:58.107003 2787 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:58.113106 2001 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.113224 2585 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.115249 2737 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:58.116852 2737 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:58.125550 1847 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.126958 2787 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:58.163570 2787 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.164899 2001 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.174221 2587 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.183446 1846 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.189636 1745 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:58.203478 1928 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:58.210376 1846 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:58.211851 2585 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:58.213610 2737 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:58.214442 2807 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:58.222995 2703 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.231904 2737 raft_consensus.cc:1062] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:22:58.240029 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9122458a1eae4f07b1a7f4ff63d6be19 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:58.240407 1846 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.250994 2585 raft_consensus.cc:2953] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.255846 2737 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:58.256755 1745 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:22:58.266119 2001 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.267657 2812 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:58.271353 2585 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.278735 2813 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:22:58.281116 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 9122458a1eae4f07b1a7f4ff63d6be19 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:22:58.282347 2805 tablet_replica.cc:331] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:22:58.283731 2805 raft_consensus.cc:2241] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:58.284550 2805 raft_consensus.cc:2270] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:58.286569 1846 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:22:58.288280 2737 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:22:58.290818 2805 ts_tablet_manager.cc:1905] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:58.311203 2737 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.312799 1846 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.320246 2001 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.340751 2587 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.341713 1745 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.350463 2805 ts_tablet_manager.cc:1918] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:58.350955 2805 log.cc:1199] T 9122458a1eae4f07b1a7f4ff63d6be19 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/9122458a1eae4f07b1a7f4ff63d6be19
I20250626 08:22:58.354158 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 9122458a1eae4f07b1a7f4ff63d6be19 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:58.380962 1920 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:22:58.385556 2587 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:58.385601 2001 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:22:58.387045 2813 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:58.387965 2812 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:22:58.393996 2812 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.397341 2587 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.398772 2001 raft_consensus.cc:2953] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.403905 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4be9d96212a6463eb1aa71d0b3d300a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:22:58.412294 1744 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:22:58.421559 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 4be9d96212a6463eb1aa71d0b3d300a6 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:22:58.423528 2819 tablet_replica.cc:331] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:22:58.424647 2819 raft_consensus.cc:2241] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:22:58.425343 2819 raft_consensus.cc:2270] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:22:58.429235 2819 ts_tablet_manager.cc:1905] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:22:58.443915 2819 ts_tablet_manager.cc:1918] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:22:58.444236 2819 log.cc:1199] T 4be9d96212a6463eb1aa71d0b3d300a6 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/4be9d96212a6463eb1aa71d0b3d300a6
I20250626 08:22:58.445747 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 4be9d96212a6463eb1aa71d0b3d300a6 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:22:59.475788 1996 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:59.483245 1928 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.483668 1847 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:22:59.484476 1965 consensus_peers.cc:487] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 856b2d0e8d2948bd9bfc33345dcd4886. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:59.485045 2820 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.485865 2798 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.493358 2799 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.498478 1847 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.506548 1928 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.524720 1744 catalog_manager.cc:5582] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.538314 2001 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:59.549278 1920 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.549293 1847 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.550848 2826 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.551981 2798 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:22:59.552898 1965 consensus_peers.cc:487] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94ba4a836fec402199d1aecf04c04b62. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:59.563944 2820 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.566128 1920 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.571280 1846 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.577644 1928 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:59.587178 1743 catalog_manager.cc:5582] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:59.593116 1890 consensus_peers.cc:487] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5329c15e45ba48438209a93439414108. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:59.611620 2001 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.623286 1847 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.623456 2813 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.624598 2835 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.632931 2812 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.684820 2001 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.685951 1846 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.692282 1996 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:22:59.693037 1745 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 08:22:59.717453 1965 consensus_peers.cc:487] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e7d84adc07945fe8a3aa16f71b111e3. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:22:59.718190 1928 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.720407 2820 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:22:59.721199 1846 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:22:59.722441 2798 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:22:59.759214 2825 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.760994 1920 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:22:59.780798 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 43699.6us
I20250626 08:22:59.780441 1847 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.787943 1745 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:22:59.978922 2847 ts_tablet_manager.cc:927] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:22:59.981798 2847 tablet_copy_client.cc:323] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:22:59.986351 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 856b2d0e8d2948bd9bfc33345dcd4886 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:22:59.986914 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 856b2d0e8d2948bd9bfc33345dcd4886 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-856b2d0e8d2948bd9bfc33345dcd4886
I20250626 08:22:59.999257 2012 tablet_copy_source_session.cc:215] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:00.002612 2847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b2d0e8d2948bd9bfc33345dcd4886. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.016253 2847 tablet_copy_client.cc:806] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:00.016844 2847 tablet_copy_client.cc:670] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:00.022472 2847 tablet_copy_client.cc:538] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:00.034183 2847 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:00.039475 2849 ts_tablet_manager.cc:927] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:00.041551 2849 tablet_copy_client.cc:323] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:00.043174 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 94ba4a836fec402199d1aecf04c04b62 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:00.043673 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 94ba4a836fec402199d1aecf04c04b62 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-94ba4a836fec402199d1aecf04c04b62
I20250626 08:23:00.063320 2012 tablet_copy_source_session.cc:215] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:00.066360 2847 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:00.067288 2847 tablet_bootstrap.cc:492] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:00.067358 2849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ba4a836fec402199d1aecf04c04b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.068030 2847 ts_tablet_manager.cc:1397] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.004s
I20250626 08:23:00.070981 2847 raft_consensus.cc:357] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.072206 2847 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:00.072849 2847 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.089753 2849 tablet_copy_client.cc:806] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:00.090443 2849 tablet_copy_client.cc:670] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:00.091907 2853 ts_tablet_manager.cc:927] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:00.094723 2849 tablet_copy_client.cc:538] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:00.095062 2853 tablet_copy_client.cc:323] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:00.096638 2847 ts_tablet_manager.cc:1428] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.028s user 0.013s sys 0.011s
I20250626 08:23:00.099309 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 7e7d84adc07945fe8a3aa16f71b111e3 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:00.099699 2011 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-856b2d0e8d2948bd9bfc33345dcd4886 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:00.100142 2011 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-856b2d0e8d2948bd9bfc33345dcd4886 on tablet 856b2d0e8d2948bd9bfc33345dcd4886 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:00.100775 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 7e7d84adc07945fe8a3aa16f71b111e3 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-7e7d84adc07945fe8a3aa16f71b111e3
I20250626 08:23:00.114462 2012 tablet_copy_source_session.cc:215] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:00.116900 2849 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:00.117422 2847 ts_tablet_manager.cc:927] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:00.120004 2853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e7d84adc07945fe8a3aa16f71b111e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.126596 2847 tablet_copy_client.cc:323] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:00.130564 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 5329c15e45ba48438209a93439414108 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:00.131112 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 5329c15e45ba48438209a93439414108 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-5329c15e45ba48438209a93439414108
I20250626 08:23:00.139710 1938 tablet_copy_source_session.cc:215] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:00.141638 2853 tablet_copy_client.cc:806] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:00.142539 2853 tablet_copy_client.cc:670] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:00.148836 2853 tablet_copy_client.cc:538] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:00.150369 2847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5329c15e45ba48438209a93439414108. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:00.157035 2853 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:00.158882 2849 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:00.159938 2849 tablet_bootstrap.cc:492] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:00.160697 2849 ts_tablet_manager.cc:1397] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.011s
I20250626 08:23:00.167520 2847 tablet_copy_client.cc:806] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:00.168131 2847 tablet_copy_client.cc:670] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:00.170647 2849 raft_consensus.cc:357] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.171594 2849 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:00.172411 2849 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.173147 2847 tablet_copy_client.cc:538] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:00.175710 2849 ts_tablet_manager.cc:1428] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250626 08:23:00.183430 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-94ba4a836fec402199d1aecf04c04b62 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:00.183918 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-94ba4a836fec402199d1aecf04c04b62 on tablet 94ba4a836fec402199d1aecf04c04b62 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:00.194589 2847 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:00.204059 2853 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:00.207398 2853 tablet_bootstrap.cc:492] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:00.207947 2853 ts_tablet_manager.cc:1397] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.002s
I20250626 08:23:00.210455 2853 raft_consensus.cc:357] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.211241 2853 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:00.211838 2853 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.214293 2853 ts_tablet_manager.cc:1428] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20250626 08:23:00.216149 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-7e7d84adc07945fe8a3aa16f71b111e3 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:00.216557 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-7e7d84adc07945fe8a3aa16f71b111e3 on tablet 7e7d84adc07945fe8a3aa16f71b111e3 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:00.216763 2847 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:00.217624 2847 tablet_bootstrap.cc:492] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:00.218297 2847 ts_tablet_manager.cc:1397] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20250626 08:23:00.220893 2847 raft_consensus.cc:357] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.221686 2847 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:00.222374 2847 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:00.224987 2847 ts_tablet_manager.cc:1428] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250626 08:23:00.226599 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-5329c15e45ba48438209a93439414108 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:23:00.226938 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-5329c15e45ba48438209a93439414108 on tablet 5329c15e45ba48438209a93439414108 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:00.452960 2585 raft_consensus.cc:1215] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:00.608362 2585 raft_consensus.cc:1215] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:00.612445 2585 raft_consensus.cc:1215] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:00.627640 2585 raft_consensus.cc:1215] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:00.989403 2820 raft_consensus.cc:1062] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:00.992853 2820 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.001183 2825 raft_consensus.cc:1062] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:01.005662 1920 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.006567 1846 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.006858 2585 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.008291 2798 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.009722 2798 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.012979 2825 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.013829 2862 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.029222 2798 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.032119 1846 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.044191 2587 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.050750 1744 catalog_manager.cc:5582] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:01.057700 1928 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.061642 2585 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.063984 1924 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.065568 1847 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.068912 2862 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.073053 2862 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.075695 2861 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.078559 2861 raft_consensus.cc:1062] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:01.097720 2861 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:01.149520 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 78188.1us
I20250626 08:23:01.166636 2812 raft_consensus.cc:1062] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:01.167061 2587 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.169450 2869 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:23:01.177239 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:01.201984 1846 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.214936 2869 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:01.222851 2812 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.224583 2586 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.228041 1924 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.221800 2820 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.246788 2868 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.290024 1846 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.291215 2586 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.291904 1848 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.295596 2839 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.289402 2798 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.300235 1996 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.301623 1846 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:01.320331 2812 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.328575 2586 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.333149 2812 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:01.332916 1743 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.342995 1924 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.345832 1920 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.367307 1745 catalog_manager.cc:5582] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.377419 2839 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.387404 1850 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.393129 2001 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.401777 1924 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.405228 2587 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.406857 2798 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.410259 2587 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.415436 1998 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.416611 2586 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.421027 2862 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.413190 1744 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.422518 1846 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.423419 2585 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.415488 2798 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.439437 1998 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.437474 2820 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20250626 08:23:01.446653 2862 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.451076 1920 raft_consensus.cc:2953] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.458510 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 856b2d0e8d2948bd9bfc33345dcd4886 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:01.464257 2868 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.466214 1846 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.443332 2001 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.469640 2585 raft_consensus.cc:2953] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.479249 1746 catalog_manager.cc:5582] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:01.507313 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 856b2d0e8d2948bd9bfc33345dcd4886 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:01.508924 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 94ba4a836fec402199d1aecf04c04b62 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:01.516234 1928 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:01.525597 2585 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.526528 1920 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.526963 2586 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.526387 1744 catalog_manager.cc:5582] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:01.527858 2867 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.528326 2876 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.530069 1846 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:01.531147 2867 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.531477 2876 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:01.540266 2839 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.541991 2862 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.543735 1920 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.550369 1846 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.551872 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5329c15e45ba48438209a93439414108 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:01.553179 2585 raft_consensus.cc:2953] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.553089 2586 raft_consensus.cc:2953] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.556327 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7e7d84adc07945fe8a3aa16f71b111e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:01.567418 2891 tablet_replica.cc:331] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:01.568523 2891 raft_consensus.cc:2241] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:01.576555 2891 raft_consensus.cc:2270] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:01.580802 2891 ts_tablet_manager.cc:1905] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:01.585162 1743 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:01.597393 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 5329c15e45ba48438209a93439414108 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:01.605487 2892 tablet_replica.cc:331] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:01.606683 2892 raft_consensus.cc:2241] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:01.607883 2892 raft_consensus.cc:2270] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:01.607949 1745 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:01.612810 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ba4a836fec402199d1aecf04c04b62 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:01.617575 2891 ts_tablet_manager.cc:1918] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:01.618196 2891 log.cc:1199] T 856b2d0e8d2948bd9bfc33345dcd4886 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/856b2d0e8d2948bd9bfc33345dcd4886
I20250626 08:23:01.620184 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 856b2d0e8d2948bd9bfc33345dcd4886 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:01.627635 2892 ts_tablet_manager.cc:1905] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:01.635413 2893 tablet_replica.cc:331] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:01.636656 2893 raft_consensus.cc:2241] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:01.639756 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e7d84adc07945fe8a3aa16f71b111e3 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:01.643913 2892 ts_tablet_manager.cc:1918] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:01.644435 2892 log.cc:1199] T 5329c15e45ba48438209a93439414108 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/5329c15e45ba48438209a93439414108
I20250626 08:23:01.641723 2893 raft_consensus.cc:2270] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:01.646858 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 5329c15e45ba48438209a93439414108 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:01.647536 2891 tablet_replica.cc:331] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:01.649821 2893 ts_tablet_manager.cc:1905] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:01.654779 2891 raft_consensus.cc:2241] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:01.655684 2891 raft_consensus.cc:2270] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:01.659395 2891 ts_tablet_manager.cc:1905] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:01.664155 2893 ts_tablet_manager.cc:1918] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:01.664613 2893 log.cc:1199] T 94ba4a836fec402199d1aecf04c04b62 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/94ba4a836fec402199d1aecf04c04b62
I20250626 08:23:01.666257 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 94ba4a836fec402199d1aecf04c04b62 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:01.673478 2891 ts_tablet_manager.cc:1918] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:01.674023 2891 log.cc:1199] T 7e7d84adc07945fe8a3aa16f71b111e3 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/7e7d84adc07945fe8a3aa16f71b111e3
I20250626 08:23:01.675488 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 7e7d84adc07945fe8a3aa16f71b111e3 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:02.654592 1846 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:02.661275 1812 consensus_peers.cc:487] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26cfef3699a141f8b7686ae2b75a3beb. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:02.662144 2001 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:02.662395 1924 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:02.663611 2846 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.664485 2845 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.674527 2846 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.675511 2001 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.692248 1924 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:02.696801 1746 catalog_manager.cc:5582] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:02.710670 1846 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:02.710750 1890 consensus_peers.cc:487] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84dfea36e4274e44b9a893967d69fd7e. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:02.711920 2001 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:02.713165 2876 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.715052 2876 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.723026 1928 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.731892 2839 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.735265 1848 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.766367 1744 catalog_manager.cc:5582] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.777331 1850 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:02.791998 1998 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:02.792403 1812 consensus_peers.cc:487] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d606b2ba0bd74eea8cd6e0c6e109c5ac. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:02.792853 1928 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:02.793824 2897 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.795984 2898 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.816046 2898 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.815485 2001 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.823395 1928 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.839811 1920 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:02.836918 2001 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.842658 1746 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:02.865491 2001 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:02.865896 1890 consensus_peers.cc:487] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0899df8ec53c45498dd1f3da415d5155. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:02.866353 1848 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:02.866816 2908 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.869691 2908 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:02.878597 2911 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.882452 2001 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.884428 1848 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:02.905893 1744 catalog_manager.cc:5582] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:03.173674 2920 ts_tablet_manager.cc:927] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:03.182988 2920 tablet_copy_client.cc:323] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:03.185490 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 84dfea36e4274e44b9a893967d69fd7e from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:03.186098 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 84dfea36e4274e44b9a893967d69fd7e from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-84dfea36e4274e44b9a893967d69fd7e
I20250626 08:23:03.194240 1938 tablet_copy_source_session.cc:215] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:03.197149 2920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84dfea36e4274e44b9a893967d69fd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:03.212185 2920 tablet_copy_client.cc:806] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:03.212787 2920 tablet_copy_client.cc:670] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:03.216861 2920 tablet_copy_client.cc:538] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:03.226876 2920 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:03.246978 2920 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:03.247855 2920 tablet_bootstrap.cc:492] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:03.248528 2920 ts_tablet_manager.cc:1397] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.002s
I20250626 08:23:03.251160 2920 raft_consensus.cc:357] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.251884 2920 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:03.252513 2920 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.256337 2920 ts_tablet_manager.cc:1428] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250626 08:23:03.255259 2923 ts_tablet_manager.cc:927] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:03.258121 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-84dfea36e4274e44b9a893967d69fd7e received from {username='slave'} at 127.0.0.1:58462
I20250626 08:23:03.258546 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-84dfea36e4274e44b9a893967d69fd7e on tablet 84dfea36e4274e44b9a893967d69fd7e with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:03.267992 2923 tablet_copy_client.cc:323] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:03.269887 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:03.270460 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet d606b2ba0bd74eea8cd6e0c6e109c5ac from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-d606b2ba0bd74eea8cd6e0c6e109c5ac
I20250626 08:23:03.277328 1861 tablet_copy_source_session.cc:215] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:03.280228 2920 ts_tablet_manager.cc:927] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:03.280890 2923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:03.282780 2920 tablet_copy_client.cc:323] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:03.284161 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 26cfef3699a141f8b7686ae2b75a3beb from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:03.284585 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 26cfef3699a141f8b7686ae2b75a3beb from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-26cfef3699a141f8b7686ae2b75a3beb
I20250626 08:23:03.293632 1861 tablet_copy_source_session.cc:215] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:03.297233 2923 tablet_copy_client.cc:806] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:03.297962 2923 tablet_copy_client.cc:670] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:03.298019 2920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26cfef3699a141f8b7686ae2b75a3beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:03.309801 2923 tablet_copy_client.cc:538] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:03.318712 2920 tablet_copy_client.cc:806] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:03.319015 2923 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:03.319561 2920 tablet_copy_client.cc:670] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:03.328145 2920 tablet_copy_client.cc:538] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:03.341773 2920 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:03.364050 2923 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:03.365000 2923 tablet_bootstrap.cc:492] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:03.365746 2923 ts_tablet_manager.cc:1397] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.047s user 0.022s sys 0.006s
I20250626 08:23:03.367100 2920 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:03.368041 2920 tablet_bootstrap.cc:492] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:03.368711 2920 ts_tablet_manager.cc:1397] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.015s
I20250626 08:23:03.368964 2923 raft_consensus.cc:357] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.369740 2923 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:03.370488 2923 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.372792 2923 ts_tablet_manager.cc:1428] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250626 08:23:03.374596 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-d606b2ba0bd74eea8cd6e0c6e109c5ac received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:03.375021 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-d606b2ba0bd74eea8cd6e0c6e109c5ac on tablet d606b2ba0bd74eea8cd6e0c6e109c5ac with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:03.371341 2920 raft_consensus.cc:357] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.376256 2920 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:03.376832 2920 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.380332 2920 ts_tablet_manager.cc:1428] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250626 08:23:03.382246 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-26cfef3699a141f8b7686ae2b75a3beb received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:03.382652 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-26cfef3699a141f8b7686ae2b75a3beb on tablet 26cfef3699a141f8b7686ae2b75a3beb with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:03.460608 2920 ts_tablet_manager.cc:927] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:03.462621 2920 tablet_copy_client.cc:323] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:03.464138 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 0899df8ec53c45498dd1f3da415d5155 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:03.464610 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 0899df8ec53c45498dd1f3da415d5155 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-0899df8ec53c45498dd1f3da415d5155
I20250626 08:23:03.470324 1938 tablet_copy_source_session.cc:215] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:03.472822 2920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0899df8ec53c45498dd1f3da415d5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:03.488288 2920 tablet_copy_client.cc:806] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:03.488819 2920 tablet_copy_client.cc:670] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:03.491838 2920 tablet_copy_client.cc:538] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:03.497967 2920 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:03.515709 2920 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:03.516422 2920 tablet_bootstrap.cc:492] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:03.517009 2920 ts_tablet_manager.cc:1397] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s
I20250626 08:23:03.519338 2920 raft_consensus.cc:357] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.519971 2920 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:03.520391 2920 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:03.522294 2920 ts_tablet_manager.cc:1428] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.005s user 0.005s sys 0.003s
I20250626 08:23:03.524856 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-0899df8ec53c45498dd1f3da415d5155 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:23:03.525266 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-0899df8ec53c45498dd1f3da415d5155 on tablet 0899df8ec53c45498dd1f3da415d5155 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:03.595259 2585 raft_consensus.cc:1215] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:03.682706 2586 raft_consensus.cc:1215] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:03.817589 2586 raft_consensus.cc:1215] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:03.851975 2586 raft_consensus.cc:1215] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:04.117763 2933 raft_consensus.cc:1062] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:04.120180 2933 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.127434 2586 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.129045 1998 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.129298 2932 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.131000 2908 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.149681 1849 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.151439 2908 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:04.150633 2932 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.151921 2001 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.157981 1850 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.171793 1746 catalog_manager.cc:5582] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.184991 2586 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.201550 2897 raft_consensus.cc:1062] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:04.204165 2897 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.216198 1998 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.217823 2931 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.218196 1920 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.223754 2587 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.230957 2931 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.233006 2931 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.263248 2897 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.264770 2587 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.268373 2898 raft_consensus.cc:1062] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:04.268812 1924 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.273948 2898 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.283754 1998 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.292796 1745 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.302093 2587 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.303411 2931 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.306910 1924 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.308672 2001 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.310284 2931 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.313079 1849 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.314262 2931 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.333689 2585 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.335603 1928 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.341646 2954 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.343964 2955 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.353826 2954 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.349762 1998 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.360227 2587 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.362443 2931 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.373103 2585 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.379194 1746 catalog_manager.cc:5582] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.385550 1928 raft_consensus.cc:2953] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.393595 2908 raft_consensus.cc:1062] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:04.396312 2908 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.409528 1924 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.419261 2001 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.420717 2938 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.422444 1850 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.452643 1745 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.455648 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:04.471500 2585 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.478212 2948 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:04.486608 2002 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.489812 2948 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.502468 2956 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.522264 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 26cfef3699a141f8b7686ae2b75a3beb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:04.523233 1850 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.523711 2585 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.528098 2938 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.550976 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet d606b2ba0bd74eea8cd6e0c6e109c5ac with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:04.551236 1998 raft_consensus.cc:2953] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.559768 1744 catalog_manager.cc:5582] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.569139 2964 tablet_replica.cc:331] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:04.570164 2585 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:04.573822 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 26cfef3699a141f8b7686ae2b75a3beb with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:04.586138 2932 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:04.586985 2964 raft_consensus.cc:2241] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:04.587783 2964 raft_consensus.cc:2270] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:04.591197 2964 ts_tablet_manager.cc:1905] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:04.592777 2908 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.600726 2965 tablet_replica.cc:331] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:04.602061 2965 raft_consensus.cc:2241] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:04.601455 1849 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.602838 2965 raft_consensus.cc:2270] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:04.609208 1928 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.615617 2965 ts_tablet_manager.cc:1905] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:04.629124 2584 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.635764 2001 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.642565 2965 ts_tablet_manager.cc:1918] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:04.643087 2965 log.cc:1199] T 26cfef3699a141f8b7686ae2b75a3beb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/26cfef3699a141f8b7686ae2b75a3beb
I20250626 08:23:04.646368 1848 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.647864 2933 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.648114 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 26cfef3699a141f8b7686ae2b75a3beb (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:04.653676 2938 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.654822 1744 catalog_manager.cc:5582] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:04.656217 2585 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.677587 2933 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.680797 2964 ts_tablet_manager.cc:1918] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:04.681244 2964 log.cc:1199] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/d606b2ba0bd74eea8cd6e0c6e109c5ac
I20250626 08:23:04.683456 1849 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.685710 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet d606b2ba0bd74eea8cd6e0c6e109c5ac (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:04.690738 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 84dfea36e4274e44b9a893967d69fd7e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:04.698305 1928 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:04.702028 2584 raft_consensus.cc:2953] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.703291 1744 catalog_manager.cc:5582] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.707263 2001 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.709017 2908 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:04.709501 2585 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:04.715430 2962 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:04.720558 2938 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.725754 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 84dfea36e4274e44b9a893967d69fd7e with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:04.726671 2964 tablet_replica.cc:331] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:04.726236 2001 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.727771 2964 raft_consensus.cc:2241] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:04.727892 2584 raft_consensus.cc:2953] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:04.730376 2964 raft_consensus.cc:2270] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:04.732179 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0899df8ec53c45498dd1f3da415d5155 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:04.743769 1744 catalog_manager.cc:5582] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:04.745373 2964 ts_tablet_manager.cc:1905] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:04.763412 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 0899df8ec53c45498dd1f3da415d5155 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:04.765689 2967 tablet_replica.cc:331] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:04.766268 2964 ts_tablet_manager.cc:1918] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:04.766710 2967 raft_consensus.cc:2241] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:04.766710 2964 log.cc:1199] T 84dfea36e4274e44b9a893967d69fd7e P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/84dfea36e4274e44b9a893967d69fd7e
I20250626 08:23:04.767624 2967 raft_consensus.cc:2270] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:04.768682 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 84dfea36e4274e44b9a893967d69fd7e (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:04.770977 2967 ts_tablet_manager.cc:1905] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:04.783370 2967 ts_tablet_manager.cc:1918] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:04.783648 2967 log.cc:1199] T 0899df8ec53c45498dd1f3da415d5155 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/0899df8ec53c45498dd1f3da415d5155
I20250626 08:23:04.785707 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 0899df8ec53c45498dd1f3da415d5155 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:05.818320 1846 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:05.831907 2001 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:05.832985 1919 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:05.833348 2948 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:05.834408 2931 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:23:05.837885 1812 consensus_peers.cc:487] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d1eb5f502d8430b9109e246276cc44e. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:05.845896 2954 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:05.851361 1919 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:05.941345 2001 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:05.951977 1996 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:05.999864 1746 catalog_manager.cc:5582] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.062208 1928 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:06.063599 2978 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:06.079192 1846 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:06.080551 1965 consensus_peers.cc:487] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75b98c18e48c4076ba1d545113f76522. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:06.082859 2978 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:06.094683 2968 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:06.127223 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 56123.4us
W20250626 08:23:06.138890 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44413.7us
I20250626 08:23:06.170969 1924 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:06.250358 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67583.9us
I20250626 08:23:06.280476 1746 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.306895 1846 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.309036 1847 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:06.338191 1812 consensus_peers.cc:487] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd7130b92c4840deaca578e523eb4a09. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:06.339874 2001 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:06.341814 2985 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:06.346715 1921 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:06.351881 2985 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:06.356666 2992 ts_tablet_manager.cc:927] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:06.383951 2992 tablet_copy_client.cc:323] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:06.386427 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 3d1eb5f502d8430b9109e246276cc44e from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:06.387116 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 3d1eb5f502d8430b9109e246276cc44e from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-3d1eb5f502d8430b9109e246276cc44e
I20250626 08:23:06.386605 2948 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.388298 1921 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.398172 2001 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.408283 1924 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:06.413476 1861 tablet_copy_source_session.cc:215] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:06.415649 1744 catalog_manager.cc:5582] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.420339 2992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1eb5f502d8430b9109e246276cc44e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 08:23:06.431252 1890 consensus_peers.cc:487] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 331d5bdb3fda41489445725e41b94bd1. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:06.431532 1847 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:06.432984 2001 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:06.434360 2908 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:06.435627 2987 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:06.444233 2992 tablet_copy_client.cc:806] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:06.444931 2992 tablet_copy_client.cc:670] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:06.452865 2992 tablet_copy_client.cc:538] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:06.454157 2908 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.457043 1847 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.462940 2992 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:06.465114 2001 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.475991 1746 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:06.495576 3000 ts_tablet_manager.cc:927] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:06.496553 2992 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:06.497624 2992 tablet_bootstrap.cc:492] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:06.498399 2992 ts_tablet_manager.cc:1397] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.009s
I20250626 08:23:06.501894 2992 raft_consensus.cc:357] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:06.503069 2992 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:06.503818 2992 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:06.505869 3000 tablet_copy_client.cc:323] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:06.508695 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 75b98c18e48c4076ba1d545113f76522 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:06.509178 2992 ts_tablet_manager.cc:1428] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250626 08:23:06.509250 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 75b98c18e48c4076ba1d545113f76522 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-75b98c18e48c4076ba1d545113f76522
I20250626 08:23:06.515581 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-3d1eb5f502d8430b9109e246276cc44e received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:06.516036 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-3d1eb5f502d8430b9109e246276cc44e on tablet 3d1eb5f502d8430b9109e246276cc44e with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:06.517073 2012 tablet_copy_source_session.cc:215] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:06.523486 3000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b98c18e48c4076ba1d545113f76522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:06.538594 3000 tablet_copy_client.cc:806] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:06.539150 3000 tablet_copy_client.cc:670] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:06.542384 3000 tablet_copy_client.cc:538] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:06.553450 3000 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:06.572109 3000 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:06.572997 3000 tablet_bootstrap.cc:492] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:06.573640 3000 ts_tablet_manager.cc:1397] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20250626 08:23:06.576105 3000 raft_consensus.cc:357] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:06.576856 3000 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:06.577473 3000 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:06.580330 3000 ts_tablet_manager.cc:1428] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250626 08:23:06.582371 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-75b98c18e48c4076ba1d545113f76522 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:06.582738 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-75b98c18e48c4076ba1d545113f76522 on tablet 75b98c18e48c4076ba1d545113f76522 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:07.269060 2584 raft_consensus.cc:1215] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:07.542106 3005 raft_consensus.cc:1062] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
W20250626 08:23:07.544241 1872 debug-util.cc:398] Leaking SignalData structure 0x7b08000c99c0 after lost signal to thread 690
W20250626 08:23:07.545560 1872 debug-util.cc:398] Leaking SignalData structure 0x7b080011bc60 after lost signal to thread 1781
W20250626 08:23:07.546993 1872 debug-util.cc:398] Leaking SignalData structure 0x7b080033c760 after lost signal to thread 1875
I20250626 08:23:07.548039 3018 ts_tablet_manager.cc:927] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
W20250626 08:23:07.548372 1872 debug-util.cc:398] Leaking SignalData structure 0x7b08002da460 after lost signal to thread 1952
W20250626 08:23:07.557246 1872 debug-util.cc:398] Leaking SignalData structure 0x7b08002338e0 after lost signal to thread 2027
W20250626 08:23:07.559000 1872 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff380 after lost signal to thread 2611
I20250626 08:23:07.568308 3018 tablet_copy_client.cc:323] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:07.578438 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet cd7130b92c4840deaca578e523eb4a09 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:07.544821 3005 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.605194 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet cd7130b92c4840deaca578e523eb4a09 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-cd7130b92c4840deaca578e523eb4a09
W20250626 08:23:07.622334 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 55382.8us
I20250626 08:23:07.622937 1744 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46332 (request call id 107) took 1030 ms. Trace:
I20250626 08:23:07.623373 1744 rpcz_store.cc:276] 0626 08:23:06.592544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:06.592701 (+ 157us) service_pool.cc:225] Handling call
0626 08:23:07.622852 (+1030151us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
W20250626 08:23:07.630059 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 111410us
W20250626 08:23:07.666599 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 46283.9us
I20250626 08:23:07.680680 1861 tablet_copy_source_session.cc:215] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:07.687542 3024 ts_tablet_manager.cc:927] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:07.691314 3024 tablet_copy_client.cc:323] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:07.691591 3007 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.692615 3007 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.693184 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 331d5bdb3fda41489445725e41b94bd1 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:07.694000 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 331d5bdb3fda41489445725e41b94bd1 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-331d5bdb3fda41489445725e41b94bd1
I20250626 08:23:07.704063 3007 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.705143 3008 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.715942 3008 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.718820 3008 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.738216 3006 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:07.739014 3006 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.740687 2585 raft_consensus.cc:1215] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:07.746524 3018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7130b92c4840deaca578e523eb4a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:07.749833 1938 tablet_copy_source_session.cc:215] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:07.762801 2987 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.763895 2987 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.764519 3008 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.765166 3008 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.766314 3006 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:07.768574 2987 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.768612 3008 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.775146 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.776895 2586 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250626 08:23:07.786566 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 45036us
W20250626 08:23:07.795469 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 49799.7us
I20250626 08:23:07.798120 3024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331d5bdb3fda41489445725e41b94bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:07.798451 2987 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.801810 3010 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.802107 2987 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.802383 3010 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.804123 2987 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.804245 3010 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.807582 3030 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:07.801291 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 69539.3us
I20250626 08:23:07.809680 3026 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:07.808163 3030 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.810333 3026 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.823009 3026 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:07.823830 3018 tablet_copy_client.cc:806] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:07.824865 3018 tablet_copy_client.cc:670] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:07.826406 2987 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.826918 2987 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.827322 3010 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.827821 3010 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.828859 2987 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:07.820398 2001 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:07.830824 3030 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:07.835023 3010 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.838270 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:07.840793 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 62192.1us
I20250626 08:23:07.844556 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:07.844965 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:07.846359 2908 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.846982 2908 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:07.849306 2908 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.849888 2908 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.850212 3027 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:07.850374 2908 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.851567 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.853490 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.853873 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.850754 3027 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.855144 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.857169 3027 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:07.858770 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.859165 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.857352 2908 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.860414 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.861284 3008 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.861938 3008 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:07.862629 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.863122 3033 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.863624 3033 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.864040 3008 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:07.865417 3010 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.865643 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.865976 3010 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.866485 3033 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.866933 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:07.867974 3010 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.867456 3033 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.868482 3010 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.868678 3033 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.870980 3033 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.871861 3024 tablet_copy_client.cc:806] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:07.872442 3024 tablet_copy_client.cc:670] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:07.871378 2908 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:07.873767 2908 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.875854 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:07.876130 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:07.875916 1893 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:07.877748 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:07.873283 3005 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250626 08:23:07.879186 3008 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.881088 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.868954 3010 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.882156 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:07.882390 3032 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.881108 2551 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:07.882967 2908 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.883056 3032 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:07.884740 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.885944 3028 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.886118 3004 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:07.876710 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.888279 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.886620 3004 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:07.889336 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.889650 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.889900 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.890934 3004 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.891321 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.891664 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.886884 3010 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.888160 3032 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.898321 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.895812 3034 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250626 08:23:07.899973 3033 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:07.887355 3024 tablet_copy_client.cc:538] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:07.901468 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.901739 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.901903 3033 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:07.895391 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.903158 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.903975 3033 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:07.904610 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.917636 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.919528 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.921110 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:07.921205 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.923327 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.923614 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.924573 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.924842 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.926004 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.926400 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:07.927839 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:07.929062 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.931164 1920 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:07.933369 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:07.936779 1814 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.935719 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:07.937958 3034 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.940183 3034 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:07.949980 2948 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:07.956427 1919 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:07.959952 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.961410 3032 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.967322 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.968608 2908 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.972957 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.974112 2908 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.974668 3024 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:07.978487 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.980329 3033 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.980779 1997 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:07.985411 1893 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.990481 3033 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:07.992028 1893 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:07.994213 1893 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:07.995862 3032 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.003353 3032 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.004607 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.013221 3032 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.006871 1744 catalog_manager.cc:5582] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.019227 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.024713 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:08.025417 3032 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.028154 1893 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.033779 2948 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.042101 3032 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.043200 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:08.044006 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:08.047053 1893 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.047761 3026 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.048103 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.049127 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.049832 3010 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.050457 3010 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.050977 3010 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.051505 3010 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.052592 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.053306 2908 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.054335 1893 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.055024 2908 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.055917 1893 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.056558 2908 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.057611 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:08.058434 2908 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:08.084157 3024 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:08.085026 3018 tablet_copy_client.cc:538] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:08.085107 3024 tablet_bootstrap.cc:492] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:08.085995 3024 ts_tablet_manager.cc:1397] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.133s user 0.020s sys 0.022s
I20250626 08:23:08.094502 2585 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.099340 3018 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:08.098856 3024 raft_consensus.cc:357] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:08.099946 3024 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:08.100600 3024 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:08.108233 3024 ts_tablet_manager.cc:1428] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20250626 08:23:08.109760 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-331d5bdb3fda41489445725e41b94bd1 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:23:08.110013 3022 raft_consensus.cc:1062] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:08.110270 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-331d5bdb3fda41489445725e41b94bd1 on tablet 331d5bdb3fda41489445725e41b94bd1 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:08.113027 3022 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:08.156599 2585 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:08.158022 3022 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:08.168787 1921 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:08.171200 3018 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:08.171460 3020 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:08.172240 3018 tablet_bootstrap.cc:492] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:08.173013 3018 ts_tablet_manager.cc:1397] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.074s user 0.021s sys 0.007s
I20250626 08:23:08.178635 1843 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:08.187696 3018 raft_consensus.cc:357] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:08.188669 3018 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:08.189371 3018 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:08.194348 3017 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.197965 3018 ts_tablet_manager.cc:1428] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.025s user 0.005s sys 0.004s
I20250626 08:23:08.204622 1921 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.209285 3022 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:08.219995 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cd7130b92c4840deaca578e523eb4a09 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:08.219090 2586 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.220440 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cd7130b92c4840deaca578e523eb4a09 on tablet cd7130b92c4840deaca578e523eb4a09 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:08.224031 1843 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:08.231238 1745 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:08.238646 1997 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:08.240258 3004 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:08.248667 3034 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.267899 2001 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.273376 2585 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:08.288465 1746 catalog_manager.cc:5582] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:08.293789 1845 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.303326 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3d1eb5f502d8430b9109e246276cc44e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:08.314639 2971 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:08.319164 2586 raft_consensus.cc:2953] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.326233 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d1eb5f502d8430b9109e246276cc44e with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:08.347189 3048 tablet_replica.cc:331] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:08.375428 3048 raft_consensus.cc:2241] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:08.376966 3048 raft_consensus.cc:2270] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:08.380770 3048 ts_tablet_manager.cc:1905] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:08.399596 3048 ts_tablet_manager.cc:1918] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:08.399966 3048 log.cc:1199] T 3d1eb5f502d8430b9109e246276cc44e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/3d1eb5f502d8430b9109e246276cc44e
I20250626 08:23:08.402405 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 3d1eb5f502d8430b9109e246276cc44e (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:08.435680 2586 raft_consensus.cc:1215] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:08.516723 2586 raft_consensus.cc:1215] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250626 08:23:08.621331 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080022c7a0 after lost signal to thread 690
W20250626 08:23:08.622356 2608 debug-util.cc:398] Leaking SignalData structure 0x7b0800229960 after lost signal to thread 1781
W20250626 08:23:08.623281 2608 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1560 after lost signal to thread 1875
W20250626 08:23:08.624269 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080031ee40 after lost signal to thread 1952
W20250626 08:23:08.625247 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080005e820 after lost signal to thread 2027
W20250626 08:23:08.626260 2608 debug-util.cc:398] Leaking SignalData structure 0x7b0800333b40 after lost signal to thread 2611
I20250626 08:23:08.759271 1997 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:08.766525 2585 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:08.771802 3020 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:08.780421 3022 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.781785 2585 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.793947 1845 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:08.795547 3020 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:08.812947 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 75b98c18e48c4076ba1d545113f76522 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:08.820693 1845 raft_consensus.cc:2953] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.834057 1745 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:08.908260 3004 raft_consensus.cc:1062] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:08.911365 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 75b98c18e48c4076ba1d545113f76522 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:08.924371 3004 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:08.946760 3032 raft_consensus.cc:1062] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:08.949594 3032 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:08.977272 3052 tablet_replica.cc:331] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:08.978705 3052 raft_consensus.cc:2241] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:08.979530 3052 raft_consensus.cc:2270] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:08.986776 3052 ts_tablet_manager.cc:1905] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:09.024686 2585 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.025331 1995 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.027297 2586 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.029201 1920 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.030185 3032 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:09.031688 3004 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:09.035300 1995 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.037051 3034 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:09.038012 3052 ts_tablet_manager.cc:1918] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:09.038475 3052 log.cc:1199] T 75b98c18e48c4076ba1d545113f76522 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/75b98c18e48c4076ba1d545113f76522
I20250626 08:23:09.040355 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 75b98c18e48c4076ba1d545113f76522 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:09.041438 1844 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:09.040261 3010 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:09.045707 3004 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:09.049360 3010 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:09.071326 3034 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.084053 3033 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.085892 2585 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.091910 2581 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.107609 1844 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.111717 1746 catalog_manager.cc:5582] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.140163 1843 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.145246 1997 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.153863 1745 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:09.178769 1920 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.189085 2585 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:09.211228 2585 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:09.213351 2001 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:09.213837 3032 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:09.219015 3010 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:09.229998 2908 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.231361 2585 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.254918 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:09.262085 2985 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:09.270557 1743 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.281001 2001 raft_consensus.cc:2953] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:09.286566 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67051.6us
I20250626 08:23:09.295919 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 331d5bdb3fda41489445725e41b94bd1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:09.296708 3004 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.305509 2585 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.321266 1746 catalog_manager.cc:5582] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.331647 1921 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.338788 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet cd7130b92c4840deaca578e523eb4a09 with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:09.345273 1919 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:09.349390 2971 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:09.356159 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 331d5bdb3fda41489445725e41b94bd1 with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:09.359023 1919 raft_consensus.cc:2953] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:09.369668 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cd7130b92c4840deaca578e523eb4a09 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:09.370097 3064 tablet_replica.cc:331] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:09.371410 3064 raft_consensus.cc:2241] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:09.372186 3064 pending_rounds.cc:70] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Trying to abort 1 pending ops.
I20250626 08:23:09.374114 3064 pending_rounds.cc:77] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7171793669773299712 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cd7130b92c4840deaca578e523eb4a09" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } } }
I20250626 08:23:09.381709 3064 raft_consensus.cc:2915] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20250626 08:23:09.382392 3064 raft_consensus.cc:2887] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250626 08:23:09.382855 3064 raft_consensus.cc:2270] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:09.390766 3064 ts_tablet_manager.cc:1905] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:09.400617 3065 tablet_replica.cc:331] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:09.401787 3065 raft_consensus.cc:2241] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:09.402457 3065 raft_consensus.cc:2270] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:09.408185 3065 ts_tablet_manager.cc:1905] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250626 08:23:09.426936 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 77297.8us
I20250626 08:23:09.494841 3064 ts_tablet_manager.cc:1918] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:09.511564 3064 log.cc:1199] T cd7130b92c4840deaca578e523eb4a09 P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/cd7130b92c4840deaca578e523eb4a09
I20250626 08:23:09.513856 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet cd7130b92c4840deaca578e523eb4a09 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:09.540916 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:09.541826 1843 raft_consensus.cc:604] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:09.574843 3065 ts_tablet_manager.cc:1918] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:09.575315 3065 log.cc:1199] T 331d5bdb3fda41489445725e41b94bd1 P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/331d5bdb3fda41489445725e41b94bd1
I20250626 08:23:09.577955 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 331d5bdb3fda41489445725e41b94bd1 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:09.579968 1919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:09.580644 1919 raft_consensus.cc:604] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:09.583384 1844 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:09.583842 1844 raft_consensus.cc:604] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:09.586572 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:09.587219 1843 raft_consensus.cc:604] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:09.589943 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:09.590965 1843 raft_consensus.cc:604] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:09.595835 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:09.596490 1843 raft_consensus.cc:604] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:09.627638 2001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dc905d51bc4a450484091ffa39379396"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:09.628360 2001 raft_consensus.cc:604] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:09.631278 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20250626 08:23:09.632997 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:09.633880 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:09.707515 3032 raft_consensus.cc:991] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.708094 3032 raft_consensus.cc:1079] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.709687 2585 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36150
I20250626 08:23:09.710404 2585 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:09.711164 2585 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.719288 2585 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.722028 2585 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:09.722719 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:09.724381 2001 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.723376 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:09.725549 1919 raft_consensus.cc:3053] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:09.725975 1919 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:09.726755 1919 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.728128 1919 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.737797 1919 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.740269 2550 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:09.741182 3028 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:09.772137 3004 raft_consensus.cc:991] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.772678 3004 raft_consensus.cc:1079] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.775557 2585 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:09.776211 2585 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:09.777052 2585 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.782169 2001 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.823596 3061 raft_consensus.cc:991] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.824079 3061 raft_consensus.cc:1079] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:09.843308 2581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:09.843940 2581 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:09.844333 2581 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.860169 2585 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.866818 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:09.867238 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:09.867647 1843 raft_consensus.cc:3053] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:09.867903 1919 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.868072 1843 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:09.869277 1843 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.879204 1843 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.866281 2585 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:09.876190 1919 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.888828 3028 raft_consensus.cc:695] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:09.888932 2550 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:09.892620 3028 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.897122 1843 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.902967 3028 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:09.903776 3028 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:09.904785 3028 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.890126 2581 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.929494 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:09.929746 2581 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:09.930207 1843 raft_consensus.cc:3053] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:09.930536 1843 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:09.931293 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:09.931941 1997 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.936475 1743 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:09.931351 1843 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.938412 1743 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:09.939643 1843 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:09.947978 1843 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.949762 1997 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:09.951187 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:09.952260 3028 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:09.952757 3028 raft_consensus.cc:695] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:09.953698 3028 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:09.988798 3061 raft_consensus.cc:991] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:09.989257 3061 raft_consensus.cc:1079] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:09.991161 1921 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:09.992187 1921 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:09.992597 1921 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.009285 3016 raft_consensus.cc:991] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:10.010196 3016 raft_consensus.cc:1079] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:10.013716 1843 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dc905d51bc4a450484091ffa39379396"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:10.014492 1843 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:10.007182 1921 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:10.014937 1843 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.018584 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:10.019228 1844 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:10.019584 1844 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:10.020795 1921 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:10.020500 1844 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:10.020988 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:10.021574 1997 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.021749 1844 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.022233 1843 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:10.024966 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:10.025473 2001 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:10.025746 2001 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:10.026412 2001 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:10.027523 2001 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.029242 1997 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:10.030680 1893 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:10.031478 3063 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:10.031934 3063 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:10.032771 3063 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:10.036557 2001 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:10.037977 1816 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:10.038913 3004 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:10.039361 3004 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:10.040282 3004 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:10.040827 1844 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:10.046207 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:10.046756 1919 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.056107 1919 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:10.059031 3066 raft_consensus.cc:991] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:10.059489 3066 raft_consensus.cc:1079] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:10.060150 1743 catalog_manager.cc:5582] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:10.061162 2583 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:10.061710 2583 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:10.062098 2583 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.039417 1843 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:10.084813 1744 catalog_manager.cc:5582] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:10.089593 1745 catalog_manager.cc:5582] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:10.094759 2583 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.096845 2583 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:10.098804 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:10.099411 2001 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.109104 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:10.109890 1843 raft_consensus.cc:3053] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:10.110371 1843 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:10.111469 1843 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.116292 1843 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.124236 3004 raft_consensus.cc:991] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:10.124747 3004 raft_consensus.cc:1079] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:10.126518 2001 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:10.127730 2581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:10.128260 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:10.130982 3028 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:10.131454 3028 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:10.132290 3028 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.129690 2581 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:10.134640 2581 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.145296 2581 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.162236 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:10.162865 1847 raft_consensus.cc:3053] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:10.163168 1847 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:10.163909 1847 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.165058 1847 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.170547 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:10.171103 1919 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:10.171738 1847 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:10.174274 2551 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:10.177178 3071 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:10.181222 1843 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:10.183547 1919 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:10.199548 2581 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:10.199757 3071 raft_consensus.cc:695] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:10.207079 1744 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:10.208870 3071 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:10.233448 1744 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:10.332118 1920 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.333791 3071 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.341789 1920 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.343163 3071 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.362193 1844 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.364380 3071 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.413586 1843 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.441030 1997 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:10.445304 3061 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:10.462306 3071 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.482534 1995 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.487712 3081 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.490787 1997 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.492882 3081 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.522749 1920 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:10.524307 3004 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:10.562708 1998 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.564179 3090 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.606830 1996 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:10.613206 2908 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:23:10.627341 3091 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250626 08:23:10.656692 1847 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.682541 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:10.736351 1844 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250626 08:23:10.783916 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 110994us
W20250626 08:23:10.786901 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 136027us
I20250626 08:23:10.791839 3104 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.833681 1996 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:10.852993 1928 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.868588 3028 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:10.871976 1928 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:10.872570 1928 raft_consensus.cc:604] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:10.892494 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 64260.2us
I20250626 08:23:10.886725 1844 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:10.912621 1921 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:10.913229 1921 raft_consensus.cc:604] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:10.915401 3090 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:10.932907 1921 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:10.933543 1921 raft_consensus.cc:604] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:10.956310 1847 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:10.958259 3110 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:10.959538 1924 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:10.960947 1965 consensus_peers.cc:487] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e74341f4bd4b68b89e1472c394f5bb. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:10.963231 3111 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:10.990324 1996 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:10.990928 1996 raft_consensus.cc:604] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:10.992103 3110 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.003569 1919 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.006052 1845 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:11.007167 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 315222us
I20250626 08:23:11.034606 1845 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:11.035245 1845 raft_consensus.cc:604] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:11.035425 1745 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.038537 2908 raft_consensus.cc:991] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.038986 2908 raft_consensus.cc:1079] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.040745 2583 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36150
I20250626 08:23:11.041330 2583 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.041718 2583 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.048353 1995 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:11.050608 1845 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:11.051223 1845 raft_consensus.cc:604] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:11.052624 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250626 08:23:11.055626 2583 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.058063 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:11.058847 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:11.058786 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:11.059427 1919 raft_consensus.cc:3053] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:11.059713 1919 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:11.060384 1919 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.070513 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:11.069598 1919 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.071151 1843 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.082772 1919 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.085863 1843 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.091550 3113 raft_consensus.cc:991] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.092024 3113 raft_consensus.cc:1079] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.092506 3016 raft_consensus.cc:991] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:11.090001 2583 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:11.093030 3022 raft_consensus.cc:1079] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:11.093622 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:11.101745 3090 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:11.102246 3090 raft_consensus.cc:695] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:11.103091 3090 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.100755 2581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:11.106083 1919 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39260
I20250626 08:23:11.107045 1919 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.107700 1919 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.116040 2581 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.116528 2581 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.129475 2581 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.132616 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:11.133240 1928 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.134500 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:11.134701 2581 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:11.135085 1844 raft_consensus.cc:3053] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:11.135396 1844 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:11.136086 1844 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.137432 1844 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.141963 1928 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.143193 2550 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:11.144186 3104 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:11.144785 3104 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:11.145747 3104 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.151317 1844 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.188644 1928 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 08:23:11.190951 1965 consensus_peers.cc:487] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e928895bcfa4d00823f3f4b50eebf86. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:11.191761 3016 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:11.195868 1844 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:11.197470 3016 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:11.154392 1919 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.154274 3109 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20250626 08:23:11.156828 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 43949us
I20250626 08:23:11.212324 1919 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:11.215027 3102 raft_consensus.cc:991] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:11.237358 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:11.238204 1844 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.236037 3107 raft_consensus.cc:1079] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:11.276293 1844 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:11.321785 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:11.322499 1999 raft_consensus.cc:3053] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:11.322763 1999 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:11.324308 1999 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.325460 1999 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.339432 1999 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:11.354857 1744 catalog_manager.cc:5582] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:11.372995 3106 raft_consensus.cc:991] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.374264 3034 raft_consensus.cc:1079] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:11.396098 2581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36142
I20250626 08:23:11.396804 2581 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.397230 2581 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.401535 1744 catalog_manager.cc:5582] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 08:23:11.418872 3124 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
W20250626 08:23:11.419804 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 195265us
I20250626 08:23:11.426465 1843 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:11.427201 1843 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.427584 1843 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.454584 1843 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.457827 1843 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:11.460425 1893 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:11.467259 2908 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:11.468919 2908 raft_consensus.cc:695] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:11.469996 2908 consensus_queue.cc:237] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.466313 1843 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:11.466423 3110 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.459041 2581 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.483336 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:11.483868 1920 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:11.484154 1920 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:11.484647 2581 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:11.484889 1920 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.486624 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:11.486930 1920 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.487164 1919 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.495918 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:11.496577 1841 raft_consensus.cc:3053] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:11.496909 1841 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:11.509388 1920 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:11.522239 1919 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.524281 1814 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:11.531091 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:11.531689 1997 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.520622 1928 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.539503 1997 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:11.497684 1841 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.539237 3061 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:11.558250 1841 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:11.603466 1744 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:11.607602 1746 catalog_manager.cc:5582] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:11.622534 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 66371.8us
I20250626 08:23:11.548907 2550 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:11.616415 3128 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250626 08:23:11.631963 1845 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:11.645774 3130 raft_consensus.cc:991] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:11.646342 3130 raft_consensus.cc:1079] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
W20250626 08:23:11.645802 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 104830us
I20250626 08:23:11.647814 3081 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:11.650446 1844 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:11.651183 1844 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:11.651602 1844 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:11.653056 3061 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:11.648896 1923 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:11.659616 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 58263.3us
I20250626 08:23:11.663437 3081 raft_consensus.cc:695] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:11.664467 3081 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:11.658815 3061 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.721798 1844 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:11.723661 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 52761.3us
I20250626 08:23:11.729442 1844 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:11.730240 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:11.731037 1928 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:11.731415 1928 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:11.732317 1928 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:11.735236 1928 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:11.736963 3138 ts_tablet_manager.cc:927] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:11.723631 3028 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250626 08:23:11.758879 1923 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:11.760998 3090 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:11.763036 1841 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:11.796406 3138 tablet_copy_client.cc:323] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:11.796959 3159 ts_tablet_manager.cc:927] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:11.803252 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 02e74341f4bd4b68b89e1472c394f5bb from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:11.803977 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 02e74341f4bd4b68b89e1472c394f5bb from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-02e74341f4bd4b68b89e1472c394f5bb
I20250626 08:23:11.806005 1928 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:11.807133 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:11.808125 3146 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:11.808581 3146 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:11.809376 3146 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:11.811618 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 139165us
I20250626 08:23:11.814944 2012 tablet_copy_source_session.cc:215] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:11.818871 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:11.817186 1744 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:11.821808 1997 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:11.829520 1997 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:11.821158 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 150426us
I20250626 08:23:11.850591 2001 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:11.850976 1923 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:11.852099 3153 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:11.852819 1841 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:11.848703 1744 catalog_manager.cc:5582] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:11.856458 3138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e74341f4bd4b68b89e1472c394f5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:11.869203 1841 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:11.871022 1848 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:11.857895 3146 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:23:11.859050 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 68719us
I20250626 08:23:11.901269 3138 tablet_copy_client.cc:806] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:11.906405 3138 tablet_copy_client.cc:670] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:11.911708 1847 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:11.875104 1744 catalog_manager.cc:5582] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:11.884979 3159 tablet_copy_client.cc:323] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:11.917981 2001 raft_consensus.cc:1273] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:11.924154 3133 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:11.927778 3138 tablet_copy_client.cc:538] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:11.927551 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 3e928895bcfa4d00823f3f4b50eebf86 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:11.928826 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 3e928895bcfa4d00823f3f4b50eebf86 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-3e928895bcfa4d00823f3f4b50eebf86
W20250626 08:23:11.948447 1812 consensus_peers.cc:487] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 148557a00636474cad8673c1f02aae5e. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:11.946532 3081 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20250626 08:23:11.939854 3090 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20250626 08:23:11.960045 1845 raft_consensus.cc:1273] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:11.965607 2012 tablet_copy_source_session.cc:215] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:12.016425 3159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e928895bcfa4d00823f3f4b50eebf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:12.031678 3138 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:11.974771 3130 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
W20250626 08:23:12.098115 3158 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.102s user 0.003s sys 0.000s
W20250626 08:23:12.108338 3166 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20250626 08:23:12.143436 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:12.117480 3155 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:12.157543 3169 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20250626 08:23:12.164829 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 83364.5us
I20250626 08:23:12.159031 1923 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:12.221159 3168 log.cc:927] Time spent T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250626 08:23:12.233315 1997 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:12.246851 1928 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:12.256091 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 77469.2us
I20250626 08:23:12.241497 3106 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250626 08:23:12.265666 3090 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:12.284427 1744 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:12.299777 1997 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:12.300275 1997 raft_consensus.cc:604] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:12.304226 1923 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:12.285480 1995 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:12.314118 3061 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:12.319728 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:12.326459 1841 raft_consensus.cc:604] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:12.339660 1997 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:12.348958 3179 ts_tablet_manager.cc:927] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:12.351578 3115 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:12.338528 3159 tablet_copy_client.cc:806] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:12.353847 2001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4f29b6256843489397f6e58f34751764"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:12.363704 3159 tablet_copy_client.cc:670] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:12.364042 2001 raft_consensus.cc:604] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:12.357245 1924 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.361124 1841 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:12.379880 1920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cca900160913464799d0cfa9d2866779"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:12.380395 1920 raft_consensus.cc:604] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:12.384815 3081 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:12.386359 3159 tablet_copy_client.cc:538] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:12.389227 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 08:23:12.390928 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:12.392971 1920 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:12.394898 3106 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:12.398306 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:12.440533 3179 tablet_copy_client.cc:323] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
W20250626 08:23:12.456014 3176 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250626 08:23:12.456578 1848 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:12.458508 3174 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:12.465173 3151 raft_consensus.cc:991] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:12.470165 3149 raft_consensus.cc:1079] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:12.497246 3138 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:12.498260 3138 tablet_bootstrap.cc:492] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:12.498946 3138 ts_tablet_manager.cc:1397] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.506s user 0.028s sys 0.000s
I20250626 08:23:12.503005 1997 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:12.503835 1841 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:12.504448 1841 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:12.504864 1841 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.507805 3159 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
W20250626 08:23:12.509464 1890 consensus_peers.cc:487] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e2fc654780843abbda82ba8f5df515e. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:12.501734 3138 raft_consensus.cc:357] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.511080 3172 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:12.513106 3061 raft_consensus.cc:991] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:12.513571 3061 raft_consensus.cc:1079] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:12.515911 1924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:12.516389 1924 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:12.516728 1924 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.519093 3138 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:12.529208 3138 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.588093 1924 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:12.596783 1924 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:12.601240 1841 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.603569 1841 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:12.604698 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:12.605315 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:12.605949 1923 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.611740 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:12.612473 2001 raft_consensus.cc:3053] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:12.612823 2001 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:12.623484 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:12.624135 1995 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.637430 1995 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:12.613690 2001 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.641039 2001 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.650525 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:12.652529 2001 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:12.701575 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 49217.3us
I20250626 08:23:12.605342 1841 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:12.706470 3174 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:12.709632 1816 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:12.722658 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 268891us
I20250626 08:23:12.737119 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 148557a00636474cad8673c1f02aae5e from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:12.737885 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 148557a00636474cad8673c1f02aae5e from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-148557a00636474cad8673c1f02aae5e
I20250626 08:23:12.730762 3157 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:12.740890 3157 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:12.741840 3157 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:12.720103 1841 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:12.743793 3174 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:12.776019 3174 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:12.768988 1923 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:12.781998 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 72798.4us
I20250626 08:23:12.782253 1924 raft_consensus.cc:1238] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:12.770483 1841 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:12.827989 1841 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.841327 3194 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:12.842675 1861 tablet_copy_source_session.cc:215] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:12.844246 1841 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:12.893651 3133 raft_consensus.cc:991] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:12.894145 3133 raft_consensus.cc:1079] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:12.910137 2585 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cca900160913464799d0cfa9d2866779"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36150
I20250626 08:23:12.910938 2585 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:12.911392 2585 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:12.960562 2585 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:12.971975 2585 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:12.979840 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 92594.4us
W20250626 08:23:12.980751 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 86094.9us
W20250626 08:23:13.002765 3191 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250626 08:23:13.010584 3138 ts_tablet_manager.cc:1428] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.511s user 0.041s sys 0.011s
W20250626 08:23:13.011014 3138 ts_tablet_manager.cc:1460] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Tablet startup took 511ms
W20250626 08:23:13.015401 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 81102.5us
W20250626 08:23:13.023800 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 43893us
W20250626 08:23:13.029361 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 47291.3us
W20250626 08:23:13.011294 3138 ts_tablet_manager.cc:1462] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Trace:
0626 08:23:11.465534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:11.465834 (+ 300us) service_pool.cc:225] Handling call
0626 08:23:11.737392 (+271558us) ts_tablet_manager.cc:928] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
0626 08:23:11.855249 (+117857us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:11.880470 (+ 25221us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:11.880665 (+ 195us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:11.900377 (+ 19712us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:11.900656 (+ 279us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:11.900941 (+ 285us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:11.900991 (+ 50us) inbound_call.cc:173] Queueing success response
0626 08:23:11.982194 (+ 81203us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0626 08:23:12.499242 (+517048us) ts_tablet_manager.cc:1429] Starting tablet replica
0626 08:23:12.499798 (+ 556us) tablet_replica.cc:260] Starting instrumentation
0626 08:23:12.500332 (+ 534us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":299,"raft.queue_time_us":138200,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":212,"tablet-copy.queue_time_us":180703,"tablet-download-pool.queue_time_us":2829,"tablet-download-pool.run_cpu_time_us":1228,"tablet-download-pool.run_wall_time_us":17895,"thread_start_us":431610,"threads_started":3}
I20250626 08:23:13.043318 2579 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:13.051412 3102 consensus_queue.cc:1046] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:13.052446 3102 raft_consensus.cc:3053] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:13.053767 3179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148557a00636474cad8673c1f02aae5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:13.058533 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-02e74341f4bd4b68b89e1472c394f5bb received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:13.059036 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-02e74341f4bd4b68b89e1472c394f5bb on tablet 02e74341f4bd4b68b89e1472c394f5bb with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:13.065164 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:13.065990 1995 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.052788 3102 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:13.079258 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 270143us
I20250626 08:23:13.090523 3148 raft_consensus.cc:991] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:13.099956 3102 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.101395 3102 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.101845 1742 catalog_manager.cc:5582] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:13.112111 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:13.110945 1845 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:13.103286 3148 raft_consensus.cc:1079] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:13.117815 3138 ts_tablet_manager.cc:927] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:13.170780 1920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4f29b6256843489397f6e58f34751764"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39260
I20250626 08:23:13.171460 1920 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:13.172071 1920 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.204007 2001 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:13.210525 1999 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:13.214589 3115 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:13.216993 1995 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:13.218221 2551 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:13.219101 3081 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:13.219609 3081 raft_consensus.cc:695] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:13.220475 3081 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.228505 1920 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.253504 1849 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:13.255548 3208 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:13.259536 1744 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:13.262413 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:13.262990 1997 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:13.263291 1997 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:13.264699 1920 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:13.263927 1997 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.266376 1997 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.268658 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:13.269272 2584 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.262869 3138 tablet_copy_client.cc:323] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:13.283721 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet 5e2fc654780843abbda82ba8f5df515e from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:13.284256 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet 5e2fc654780843abbda82ba8f5df515e from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-5e2fc654780843abbda82ba8f5df515e
I20250626 08:23:13.289613 1745 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:13.292008 1938 tablet_copy_source_session.cc:215] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:13.309245 3138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2fc654780843abbda82ba8f5df515e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:13.310653 1928 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:13.313792 1923 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:13.320837 2585 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Advancing to term 2
I20250626 08:23:13.338840 1997 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:13.330472 3205 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20250626 08:23:13.353493 1893 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:13.362488 2584 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:13.363647 3102 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:13.375093 2585 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:13.378907 3102 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:13.407980 1997 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:13.381284 3142 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20250626 08:23:13.410357 3130 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250626 08:23:13.483587 3102 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.501466 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:13.500370 1742 catalog_manager.cc:5582] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:13.593844 1997 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:13.594537 1997 raft_consensus.cc:604] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:13.605422 1928 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5329c15e45ba48438209a93439414108"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:13.606050 1928 raft_consensus.cc:604] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:13.611338 3159 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:13.618965 1997 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:13.619755 1997 raft_consensus.cc:604] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:13.622915 3142 raft_consensus.cc:1062] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:13.688333 3159 tablet_bootstrap.cc:492] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:13.689428 3159 ts_tablet_manager.cc:1397] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 1.182s user 0.033s sys 0.043s
I20250626 08:23:13.700510 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:13.720038 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:13.720849 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:13.704015 3207 raft_consensus.cc:991] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:13.738715 3149 raft_consensus.cc:1079] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:13.693820 3179 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36142 (request call id 325) took 1379 ms. Trace:
I20250626 08:23:13.843211 3179 rpcz_store.cc:276] 0626 08:23:12.314730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:12.314867 (+ 137us) service_pool.cc:225] Handling call
0626 08:23:12.362965 (+ 48098us) ts_tablet_manager.cc:928] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
0626 08:23:13.052471 (+689506us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:13.285883 (+233412us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:13.286148 (+ 265us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:13.692875 (+406727us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:13.693246 (+ 371us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:13.693622 (+ 376us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:13.693715 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":23421,"tablet-copy.queue_time_us":33577,"thread_start_us":111682,"threads_started":2}
I20250626 08:23:13.844791 3179 tablet_copy_client.cc:806] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:13.844709 3159 raft_consensus.cc:357] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:13.845543 3179 tablet_copy_client.cc:670] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:13.766942 3142 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:13.845832 3159 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:13.858597 3159 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:13.860680 3159 ts_tablet_manager.cc:1428] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.171s user 0.011s sys 0.003s
W20250626 08:23:13.865375 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 77308.5us
I20250626 08:23:13.871065 3150 raft_consensus.cc:991] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:13.871523 3150 raft_consensus.cc:1079] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:13.872180 3138 tablet_copy_client.cc:806] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:13.872784 3138 tablet_copy_client.cc:670] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
W20250626 08:23:13.879091 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 125112us
I20250626 08:23:13.908627 1841 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:13.909474 1841 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:13.909986 1841 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.934039 1999 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:13.951057 1997 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:13.952379 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-3e928895bcfa4d00823f3f4b50eebf86 received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:13.968039 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-3e928895bcfa4d00823f3f4b50eebf86 on tablet 3e928895bcfa4d00823f3f4b50eebf86 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:13.960927 2584 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:13.976984 1928 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39260
I20250626 08:23:13.977656 1928 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:13.978150 1928 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:13.953171 3153 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
W20250626 08:23:13.977411 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 240804us
I20250626 08:23:13.981361 3198 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:13.987633 1923 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:13.999358 1928 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.002581 1928 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:14.004815 1920 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:14.009133 3203 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:14.017736 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:14.018355 2585 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.022089 1841 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:14.026298 2585 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:14.026777 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:14.027345 1995 raft_consensus.cc:3053] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:14.027644 1995 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:14.028558 1995 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.030123 1995 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.031095 1997 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:14.034118 3138 tablet_copy_client.cc:538] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:14.035666 1890 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:14.036298 1995 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:14.039188 2585 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:14.045250 3226 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:14.046627 3102 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:14.047144 3198 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250626 08:23:14.045719 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 90239.5us
W20250626 08:23:14.052443 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:14.061301 3102 raft_consensus.cc:695] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:14.062264 3102 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.058987 3227 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250626 08:23:14.106230 2585 raft_consensus.cc:1215] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:14.112957 1746 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:14.093294 3142 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:14.143273 1845 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:14.143879 1845 raft_consensus.cc:604] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership
I20250626 08:23:14.168391 3175 raft_consensus.cc:991] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
W20250626 08:23:14.212240 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 46434.1us
I20250626 08:23:14.202819 1845 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:14.219659 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:14.221275 1997 raft_consensus.cc:3053] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:14.248803 1997 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:14.211061 3115 raft_consensus.cc:991] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:14.254320 3146 raft_consensus.cc:1079] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:14.255301 3115 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:14.256474 1999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:14.244866 3207 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20250626 08:23:14.246109 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 109768us
I20250626 08:23:14.204813 3179 tablet_copy_client.cc:538] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:14.226707 1998 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:14.264721 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:14.265331 1923 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.201938 1841 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:14.273483 3138 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:14.266814 3225 raft_consensus.cc:1079] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:14.282922 1999 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:14.283372 1999 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:14.300840 1999 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:14.306649 3232 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.115s user 0.003s sys 0.000s
I20250626 08:23:14.288749 1841 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5329c15e45ba48438209a93439414108"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:14.320470 1841 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:14.321028 1841 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.321575 1746 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:14.329141 1923 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:14.330840 3235 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20250626 08:23:14.329896 1999 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:14.340219 1997 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:14.341215 1997 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.342635 2585 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:14.344630 3179 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:14.348105 1997 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:14.348626 3207 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250626 08:23:14.350320 1848 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:14.350914 1848 raft_consensus.cc:604] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership
W20250626 08:23:14.356334 1733 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 02e74341f4bd4b68b89e1472c394f5bb already in progress [suppressed 2 similar messages]
I20250626 08:23:14.357482 1816 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:14.374303 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 73032.2us
W20250626 08:23:14.375346 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67549.2us
I20250626 08:23:14.376766 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:14.377144 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:14.377358 2585 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:14.377712 1845 raft_consensus.cc:3053] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:14.378082 1845 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:14.378700 1845 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.379640 1845 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:14.382323 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 98850.4us
I20250626 08:23:14.385455 1845 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:14.386142 2585 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:14.391355 1919 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:14.355950 1841 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.396113 1841 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:14.406397 1733 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 50102.9us
I20250626 08:23:14.409088 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:14.409673 2585 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.413547 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:14.414162 1919 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:14.414458 1919 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:14.415145 1919 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.415580 2585 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:14.416373 1919 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:14.416703 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:14.387701 3153 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
W20250626 08:23:14.489758 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 73143us
W20250626 08:23:14.490612 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 81080.6us
W20250626 08:23:14.487939 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 65592.6us
W20250626 08:23:14.485308 3219 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20250626 08:23:14.496582 3146 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:14.507217 1919 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:14.510531 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 121028us
I20250626 08:23:14.515946 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:14.516516 1921 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:14.513247 1968 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:14.537499 1921 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:14.517494 3146 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:14.532464 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:14.585206 1745 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:14.541471 3207 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
W20250626 08:23:14.602468 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 105896us
I20250626 08:23:14.610245 3146 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.583652 3153 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:14.614948 3153 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:14.588013 1841 raft_consensus.cc:604] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Received request to transfer leadership
I20250626 08:23:14.627625 1841 raft_consensus.cc:612] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250626 08:23:14.642988 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 100297us
W20250626 08:23:14.643585 1795 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20250626 08:23:14.643942 1795 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20250626 08:23:14.618957 3207 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:23:14.654992 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 51386.2us
W20250626 08:23:14.695204 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 47909.2us
I20250626 08:23:14.707792 2585 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:14.765781 1996 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:14.748163 3241 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250626 08:23:14.752020 3207 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:14.831591 3174 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250626 08:23:14.853483 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 153420us
W20250626 08:23:14.865278 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 68066us
W20250626 08:23:14.842072 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 80476us
I20250626 08:23:14.873291 1745 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:14.898700 3151 raft_consensus.cc:1062] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
W20250626 08:23:14.958863 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:14.980178 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 294552us
I20250626 08:23:14.981753 3151 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:15.047111 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 349403us
I20250626 08:23:15.055550 1920 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:15.007216 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:15.076045 3207 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:15.076795 1742 catalog_manager.cc:5582] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:15.079237 1742 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:15.083109 2584 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:15.096546 1843 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:15.103780 3100 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250626 08:23:15.115052 1925 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:15.122167 3115 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:15.138464 3207 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20250626 08:23:15.176079 1925 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:15.178771 2579 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:15.181385 3115 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:15.196125 2002 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:15.201109 3262 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250626 08:23:15.201635 3143 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250626 08:23:15.216609 3138 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:15.217602 3138 tablet_bootstrap.cc:492] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:15.218340 3138 ts_tablet_manager.cc:1397] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.945s user 0.031s sys 0.030s
I20250626 08:23:15.221365 3149 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:15.206629 1920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:15.223161 2580 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:15.329558 1920 raft_consensus.cc:604] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:15.226208 3179 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:15.337978 3179 tablet_bootstrap.cc:492] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:15.338697 3179 ts_tablet_manager.cc:1397] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.994s user 0.048s sys 0.034s
I20250626 08:23:15.348613 1844 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:15.353771 1920 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:15.363865 3226 raft_consensus.cc:991] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:15.364117 1928 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:15.364379 3226 raft_consensus.cc:1079] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:15.364679 1928 raft_consensus.cc:604] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:15.373528 1843 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:15.374145 1843 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:15.374512 1843 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:15.385156 1928 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:15.385843 1928 raft_consensus.cc:604] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:15.378063 3100 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20250626 08:23:15.390806 1843 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:15.392246 1928 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:15.392601 1843 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:15.395395 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8353d62d68914958b13bb87541c357f8"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:15.396602 1843 raft_consensus.cc:604] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:15.399257 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:15.402411 1928 raft_consensus.cc:3053] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:15.403434 1928 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:15.383939 1745 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:15.408138 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:15.408671 1999 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:15.418119 1928 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:15.419037 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:15.419726 1843 raft_consensus.cc:604] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:15.426364 1999 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:15.428390 1928 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:15.436360 1816 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:15.533967 3275 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
W20250626 08:23:15.544166 3274 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20250626 08:23:15.548952 3138 raft_consensus.cc:357] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:15.551674 3272 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
I20250626 08:23:15.434306 3179 raft_consensus.cc:357] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:15.548426 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 112304us
I20250626 08:23:15.415184 3101 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250626 08:23:15.547899 1928 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:15.552264 3203 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
W20250626 08:23:15.610795 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 54058.3us
I20250626 08:23:15.611565 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250626 08:23:15.558795 3138 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:15.613207 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:15.622305 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:23:15.632444 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44035.6us
I20250626 08:23:15.569497 3179 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:15.632802 3203 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:15.757653 3203 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:15.827358 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 194198us
W20250626 08:23:15.824437 3276 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20250626 08:23:15.732224 3179 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:15.886742 2587 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250626 08:23:15.910542 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 289246us
I20250626 08:23:15.821527 3138 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:15.923408 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 91942.2us
I20250626 08:23:15.932816 3179 ts_tablet_manager.cc:1428] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.576s user 0.012s sys 0.006s
W20250626 08:23:15.933228 3179 ts_tablet_manager.cc:1460] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Tablet startup took 576ms
I20250626 08:23:15.930624 1843 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:15.946004 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 104535us
W20250626 08:23:15.943783 2580 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.525s user 0.009s sys 0.004s
W20250626 08:23:15.968294 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 93120.5us
W20250626 08:23:15.993326 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 81296.1us
I20250626 08:23:15.995548 3138 ts_tablet_manager.cc:1428] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.777s user 0.011s sys 0.008s
W20250626 08:23:15.995996 3138 ts_tablet_manager.cc:1460] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Tablet startup took 777ms
W20250626 08:23:15.933535 3179 ts_tablet_manager.cc:1462] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Trace:
0626 08:23:12.314730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:12.314867 (+ 137us) service_pool.cc:225] Handling call
0626 08:23:12.362965 (+ 48098us) ts_tablet_manager.cc:928] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
0626 08:23:13.052471 (+689506us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:13.285883 (+233412us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:13.286148 (+ 265us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:13.692875 (+406727us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:13.693246 (+ 371us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:13.693622 (+ 376us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:13.693715 (+ 93us) inbound_call.cc:173] Queueing success response
0626 08:23:14.344263 (+650548us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0626 08:23:15.356713 (+1012450us) ts_tablet_manager.cc:1429] Starting tablet replica
0626 08:23:15.357241 (+ 528us) tablet_replica.cc:260] Starting instrumentation
0626 08:23:15.357807 (+ 566us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":23421,"mutex_wait_us":7427,"raft.queue_time_us":15497,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3528,"tablet-copy.queue_time_us":33577,"tablet-download-pool.queue_time_us":150,"tablet-download-pool.run_cpu_time_us":1103,"tablet-download-pool.run_wall_time_us":358538,"thread_start_us":111682,"threads_started":2}
W20250626 08:23:16.036741 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 76249us
W20250626 08:23:16.049667 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 76441.9us
W20250626 08:23:15.996307 3138 ts_tablet_manager.cc:1462] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Trace:
0626 08:23:13.110139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:13.115726 (+ 5587us) service_pool.cc:225] Handling call
0626 08:23:13.121322 (+ 5596us) ts_tablet_manager.cc:928] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
0626 08:23:13.308076 (+186754us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:13.497313 (+189237us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:13.497519 (+ 206us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:13.871239 (+373720us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:13.871525 (+ 286us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:13.871850 (+ 325us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:13.871907 (+ 57us) inbound_call.cc:173] Queueing success response
0626 08:23:14.273108 (+401201us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0626 08:23:15.218621 (+945513us) ts_tablet_manager.cc:1429] Starting tablet replica
0626 08:23:15.378727 (+160106us) tablet_replica.cc:260] Starting instrumentation
0626 08:23:15.410993 (+ 32266us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":10617,"mutex_wait_us":15406,"raft.queue_time_us":12026,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":11456,"tablet-copy.queue_time_us":1636,"tablet-download-pool.queue_time_us":117,"tablet-download-pool.run_cpu_time_us":1347,"tablet-download-pool.run_wall_time_us":160677,"thread_start_us":88947,"threads_started":1}
W20250626 08:23:16.066094 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 105538us
I20250626 08:23:16.066617 3134 raft_consensus.cc:991] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.085708 3134 raft_consensus.cc:1079] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.086802 3304 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20250626 08:23:16.094030 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 53848.9us
W20250626 08:23:16.107227 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 92748.6us
I20250626 08:23:16.113996 2585 raft_consensus.cc:1215] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:16.144498 1998 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34656
I20250626 08:23:16.145025 1998 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:16.145362 1998 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.148312 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-148557a00636474cad8673c1f02aae5e received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:16.197588 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-148557a00636474cad8673c1f02aae5e on tablet 148557a00636474cad8673c1f02aae5e with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:16.155725 3134 raft_consensus.cc:991] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:16.198973 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-5e2fc654780843abbda82ba8f5df515e received from {username='slave'} at 127.0.0.1:58462
W20250626 08:23:16.187012 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 142097us
I20250626 08:23:16.204389 1998 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.198522 3134 raft_consensus.cc:1079] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:16.199402 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-5e2fc654780843abbda82ba8f5df515e on tablet 5e2fc654780843abbda82ba8f5df515e with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:16.244705 1998 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:16.246762 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:16.247579 2585 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.304041 1845 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:16.304838 1845 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:16.305240 1845 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:16.308388 3286 raft_consensus.cc:991] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.319578 3143 raft_consensus.cc:1079] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.326982 1999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8353d62d68914958b13bb87541c357f8"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:16.327653 1999 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:16.328652 1999 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.345966 1744 catalog_manager.cc:5582] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:16.375242 3174 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:16.377878 3290 raft_consensus.cc:991] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.378100 3174 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.378774 3288 raft_consensus.cc:1079] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:16.383395 3174 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:16.394541 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:16.387143 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:16.411427 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:16.420822 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:16.427299 2002 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:16.428962 2002 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:16.429327 2002 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.440786 2585 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:16.442440 1928 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250626 08:23:16.445851 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:16.448297 1999 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.449642 3306 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:16.447484 1965 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:16.453578 3287 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:16.456547 1999 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:16.457500 2584 raft_consensus.cc:1215] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:16.459263 3287 raft_consensus.cc:695] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:16.460282 3287 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.463912 1845 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:16.464838 3286 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:16.484654 2585 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:16.496094 2002 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:16.466252 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:16.469163 1843 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:16.472370 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:16.531935 1996 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:16.480060 1998 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.482262 1845 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:16.515322 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:16.524533 2002 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:16.533241 1848 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:16.533349 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:16.533610 1848 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:16.534004 1920 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:16.534334 1920 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:16.534968 1920 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:16.536042 1920 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:16.534554 1848 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.538460 1848 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.540621 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:16.541167 1928 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.547618 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:16.548440 1999 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:16.551883 1996 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:16.560779 3205 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:16.539829 2584 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:16.572736 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 45908.9us
I20250626 08:23:16.556998 1918 raft_consensus.cc:3053] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:16.578696 2908 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:16.583580 3142 consensus_queue.cc:1046] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:16.592072 2002 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:16.593096 2584 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:16.584245 1918 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:16.599326 3290 raft_consensus.cc:3053] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:16.599895 3290 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:16.599400 1918 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.600718 1918 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.600682 3290 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:16.601959 3290 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:16.603026 1928 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:16.605741 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:16.617659 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:16.618357 3281 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:16.619477 3281 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:16.619659 3304 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:16.620240 3304 raft_consensus.cc:695] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:16.621009 3304 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:16.620754 3281 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:16.632122 1999 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:16.636479 2584 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:16.699012 1848 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:16.703004 1842 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:16.705507 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:16.738569 1927 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:16.740242 1920 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:16.742455 1918 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:16.746358 1814 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:16.755498 2002 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
W20250626 08:23:16.770210 2908 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250626 08:23:16.790364 3267 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:16.791098 3267 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:16.792366 3267 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:16.872014 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 90005.6us
I20250626 08:23:16.894285 3201 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20250626 08:23:16.931702 3149 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
W20250626 08:23:16.943126 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 109082us
W20250626 08:23:16.975636 3328 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.134s user 0.002s sys 0.002s
I20250626 08:23:16.975992 2580 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:16.993482 3260 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:17.001487 1742 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:17.007865 3149 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.011056 2582 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.034091 1847 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:17.039536 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46432.7us
I20250626 08:23:17.065418 3335 raft_consensus.cc:1062] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:17.069226 1928 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:17.074137 3260 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:17.082666 3288 raft_consensus.cc:1062] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:17.068589 3335 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.114825 1850 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:17.141577 3148 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:17.149629 1847 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:17.153473 3259 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:17.105226 3288 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.169755 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:17.178707 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3e928895bcfa4d00823f3f4b50eebf86 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:17.190946 1746 catalog_manager.cc:5582] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:17.200227 2585 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.201122 1999 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.209892 3278 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250626 08:23:17.206308 1847 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.215066 2908 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:17.228111 3335 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250626 08:23:17.239893 1849 raft_consensus.cc:2953] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.249003 1928 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:17.256357 3148 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:17.270447 2582 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:17.274186 3303 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:17.319291 1928 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:17.325282 3310 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250626 08:23:17.348464 3335 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.357173 1847 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.384516 1998 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:17.386204 3310 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:17.390216 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:17.391060 1849 raft_consensus.cc:604] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:17.394176 1928 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:17.394862 1928 raft_consensus.cc:604] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:17.395238 1928 raft_consensus.cc:612] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:23:17.394362 1999 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.410059 1847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:17.410636 1847 raft_consensus.cc:604] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:17.417704 3336 raft_consensus.cc:991] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:17.418977 3336 raft_consensus.cc:1079] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:17.421128 1999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:17.421665 2585 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.419018 2582 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.426419 1998 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.430073 3267 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:17.421701 1999 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:17.431299 1999 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:17.445631 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:17.447079 1849 raft_consensus.cc:604] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:17.447527 1849 raft_consensus.cc:612] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:23:17.440652 3310 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250626 08:23:17.450352 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 08:23:17.452014 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:17.453075 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:17.463428 1999 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:17.485481 1918 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:17.491400 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:17.492620 1924 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:17.490726 1999 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:17.492182 3336 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20250626 08:23:17.501881 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:17.502574 1849 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:17.503068 1849 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:17.537974 1849 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:17.539304 1849 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:17.542676 1924 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:17.552665 1967 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:17.553807 3260 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:17.554327 3260 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:17.555269 3260 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:17.560240 1849 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:17.587059 3336 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.590859 1746 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:17.596180 1746 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:23:17.603195 3278 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:17.613808 3278 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.616294 3278 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:17.617085 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:17.617417 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:17.618737 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:17.619940 1893 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:17.621980 3278 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:17.621423 1997 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.632046 1918 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.648444 2582 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.649190 1744 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.653824 1924 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.653512 1744 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.689069 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e928895bcfa4d00823f3f4b50eebf86 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:17.696811 1997 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.716596 2582 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:17.718092 3335 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:17.727017 3226 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.730710 3336 raft_consensus.cc:991] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:17.731244 3336 raft_consensus.cc:1079] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:17.734236 1843 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:17.736048 3278 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:17.736320 2002 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:17.736826 2002 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:17.737170 2002 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:17.731765 2582 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.753933 1841 raft_consensus.cc:2953] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.762576 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5e2fc654780843abbda82ba8f5df515e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:17.768167 2002 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.772336 1741 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:17.774075 2582 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:17.776101 3148 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:17.777338 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:17.777902 2582 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:17.780308 1924 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:17.780846 2002 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:17.782030 3260 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:17.782929 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:17.783478 1843 raft_consensus.cc:3053] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:17.783776 1843 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:17.784430 1843 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.784520 3260 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.786949 2582 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:17.786378 2585 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.788601 1965 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:17.789434 3259 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:17.790091 3259 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:17.790998 3259 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.789515 3362 tablet_replica.cc:331] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:17.796312 3362 raft_consensus.cc:2241] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:17.789402 1843 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:17.797485 1924 raft_consensus.cc:2953] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.797323 3362 raft_consensus.cc:2270] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:17.805804 3362 ts_tablet_manager.cc:1905] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:17.827500 1742 catalog_manager.cc:5582] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:17.838527 1843 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:17.854038 1743 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:17.855903 1743 catalog_manager.cc:5582] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 4 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.865388 3362 ts_tablet_manager.cc:1918] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:17.866134 3362 log.cc:1199] T 3e928895bcfa4d00823f3f4b50eebf86 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/3e928895bcfa4d00823f3f4b50eebf86
I20250626 08:23:17.866411 1847 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:17.868350 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 3e928895bcfa4d00823f3f4b50eebf86 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:17.872658 1920 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:17.874382 3301 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:17.867225 1741 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:17.883124 3336 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.888914 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2fc654780843abbda82ba8f5df515e with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:17.891510 1920 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.896068 2585 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:17.899392 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 02e74341f4bd4b68b89e1472c394f5bb with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:17.900947 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 148557a00636474cad8673c1f02aae5e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:17.907327 3301 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:17.906848 3365 tablet_replica.cc:331] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:17.919221 1743 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:17.923641 3365 raft_consensus.cc:2241] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:17.928314 3365 raft_consensus.cc:2270] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:17.941845 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 02e74341f4bd4b68b89e1472c394f5bb with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:17.942842 1989 tablet_service.cc:1515] Processing DeleteTablet for tablet 148557a00636474cad8673c1f02aae5e with delete_type TABLET_DATA_TOMBSTONED (TS 755d83ca0913406b816e95415d502055 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34630
I20250626 08:23:17.948882 2585 raft_consensus.cc:2953] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 755d83ca0913406b816e95415d502055 (127.0.171.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:17.963558 3366 tablet_replica.cc:331] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:17.968329 3366 raft_consensus.cc:2241] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:17.969238 3366 raft_consensus.cc:2270] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:17.973861 3366 ts_tablet_manager.cc:1905] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:17.970590 3365 ts_tablet_manager.cc:1905] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:18.001077 3366 ts_tablet_manager.cc:1918] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250626 08:23:18.001657 3366 log.cc:1199] T 02e74341f4bd4b68b89e1472c394f5bb P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/02e74341f4bd4b68b89e1472c394f5bb
I20250626 08:23:18.004429 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 02e74341f4bd4b68b89e1472c394f5bb (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:18.006505 3365 ts_tablet_manager.cc:1918] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:18.007045 3365 log.cc:1199] T 5e2fc654780843abbda82ba8f5df515e P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/5e2fc654780843abbda82ba8f5df515e
I20250626 08:23:18.008672 3365 tablet_replica.cc:331] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: stopping tablet replica
I20250626 08:23:18.008620 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 5e2fc654780843abbda82ba8f5df515e (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:18.009820 3365 raft_consensus.cc:2241] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:18.010725 3365 raft_consensus.cc:2270] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:18.017760 3365 ts_tablet_manager.cc:1905] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:18.035979 3365 ts_tablet_manager.cc:1918] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:18.036520 3365 log.cc:1199] T 148557a00636474cad8673c1f02aae5e P 755d83ca0913406b816e95415d502055: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-2-root/wals/148557a00636474cad8673c1f02aae5e
I20250626 08:23:18.038224 1733 catalog_manager.cc:4928] TS 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): tablet 148557a00636474cad8673c1f02aae5e (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:18.144618 1920 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:18.145882 1843 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:18.146101 3259 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:18.147950 3260 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:18.331787 1849 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:18.333734 3259 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:18.360018 2585 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:18.361546 3297 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:18.664026 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:18.729506 2002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:18.731637 2002 raft_consensus.cc:604] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:18.734658 2002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:18.735271 2002 raft_consensus.cc:604] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:18.742790 2002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:18.743467 2002 raft_consensus.cc:604] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:18.747262 2002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:18.748062 2002 raft_consensus.cc:604] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:18.753764 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5329c15e45ba48438209a93439414108"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:18.754355 1849 raft_consensus.cc:604] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:18.755853 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250626 08:23:18.757555 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:18.758333 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:18.779703 3301 raft_consensus.cc:991] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:18.780293 3301 raft_consensus.cc:1079] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:18.781958 1920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5329c15e45ba48438209a93439414108"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:18.782496 1920 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:18.782864 1920 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:18.793813 3259 raft_consensus.cc:991] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:18.794380 3259 raft_consensus.cc:1079] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:18.796687 1849 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:18.797704 1849 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:18.798115 1849 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.804018 1920 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.806618 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:18.806633 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:18.807354 1841 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:18.807365 2582 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:18.807881 1841 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:18.809679 1841 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.812852 1841 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:18.820603 2582 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:18.821084 1920 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:18.830560 1890 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:18.824424 1849 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:18.832199 3332 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:18.832724 3332 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:18.833002 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:18.833588 1920 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.833688 3332 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.839195 1849 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:18.842670 1920 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:18.844097 1814 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:18.845983 3283 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:18.846539 3283 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:18.849220 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:18.849812 1997 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:18.850179 1997 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:18.850879 1997 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:18.847470 3283 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:18.854633 1997 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.882825 3303 raft_consensus.cc:991] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:18.883515 3303 raft_consensus.cc:1079] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:18.907284 1741 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:18.911417 1746 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:18.895991 1841 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:18.923512 1924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39260
I20250626 08:23:18.924165 1924 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:18.924587 1924 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.926429 1997 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:18.950989 1924 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.952921 1924 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:18.955838 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:18.959401 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:18.960017 2585 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.956456 1997 raft_consensus.cc:3053] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:18.960493 1997 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:18.965411 1997 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.966957 1997 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:18.976263 2585 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:18.978056 1890 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:18.978976 3226 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:18.979693 3226 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:18.980628 3226 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:18.985426 1997 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:19.025642 1746 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:19.035974 3303 raft_consensus.cc:991] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:19.036485 3303 raft_consensus.cc:1079] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:19.061873 3259 raft_consensus.cc:991] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:19.062443 3259 raft_consensus.cc:1079] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:19.068526 1841 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:19.069054 1841 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:19.069379 1841 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.038959 1924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39260
I20250626 08:23:19.081095 1924 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:19.081601 1924 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.099794 1841 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:19.103681 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:19.104207 1920 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.115350 1843 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.120056 1920 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:19.122121 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:19.122308 1814 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:19.122663 2002 raft_consensus.cc:3053] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:19.122982 2002 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:19.123503 3301 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:19.124037 3301 raft_consensus.cc:695] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:19.123740 2002 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:19.126051 2002 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.126586 1841 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:19.128890 1924 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:19.124854 3301 consensus_queue.cc:237] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:19.131734 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:19.132321 1841 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.140100 1924 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:19.142342 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:19.142911 1997 raft_consensus.cc:3053] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:19.143215 1997 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:19.143937 1997 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:19.145076 1997 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:19.148214 1924 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:19.149106 1999 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:19.154181 3262 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:19.156801 1841 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:19.160821 1893 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:19.157080 3283 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.161701 3226 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:19.162214 3226 raft_consensus.cc:695] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:19.163033 3226 consensus_queue.cc:237] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:19.165666 1812 consensus_peers.cc:487] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cef3ff80065343cdbaf58ef8b1bbedf9. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:19.190013 2002 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:19.191303 1746 catalog_manager.cc:5582] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:19.193022 1997 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:19.204772 3283 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.217450 2002 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.246739 1924 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.247907 1746 catalog_manager.cc:5582] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:19.262950 1841 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:19.264614 3332 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:19.267570 1996 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.269101 3262 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.331801 2585 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:19.333164 3278 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:19.341435 1920 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.343780 3336 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.377708 1997 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.381875 2582 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:19.395931 1741 catalog_manager.cc:5582] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:19.436105 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 49133.1us
I20250626 08:23:19.444391 1928 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250626 08:23:19.445564 1965 consensus_peers.cc:487] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a71e0323b96e4a47b1ba2e0a3d72840f. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:19.446007 3297 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:19.451376 1841 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:19.456574 3395 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:19.453331 3149 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250626 08:23:19.545157 1841 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.548094 1999 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:19.527388 3259 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.558530 3398 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:19.568593 1741 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.584326 1920 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.600384 1997 raft_consensus.cc:1273] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.602610 3283 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.619273 1841 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.630425 2002 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:19.631937 1920 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:19.631999 3262 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250626 08:23:19.635152 3283 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:19.645038 1928 raft_consensus.cc:1273] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.647346 3388 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.649286 3336 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.656822 2002 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.663173 1924 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
W20250626 08:23:19.669976 1812 consensus_peers.cc:487] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 171f89014c824d17a510a3cc17b3af0d. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:19.688856 1843 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.690194 3398 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.695336 3408 ts_tablet_manager.cc:927] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:19.703559 1999 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:19.706529 3398 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:19.745362 1743 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 3, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.764609 1997 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.770613 3408 tablet_copy_client.cc:323] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:19.778113 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet cef3ff80065343cdbaf58ef8b1bbedf9 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:19.778717 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet cef3ff80065343cdbaf58ef8b1bbedf9 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-cef3ff80065343cdbaf58ef8b1bbedf9
I20250626 08:23:19.793147 1861 tablet_copy_source_session.cc:215] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:19.808215 3408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef3ff80065343cdbaf58ef8b1bbedf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:19.844692 3408 tablet_copy_client.cc:806] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:19.846279 3408 tablet_copy_client.cc:670] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:19.851816 3408 tablet_copy_client.cc:538] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250626 08:23:19.857999 1965 consensus_peers.cc:487] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ce63b4769e4ca8980ddb20229389ef. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:19.858333 1920 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:19.860164 3297 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.861964 1843 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:19.865935 3259 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:19.873101 3408 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:19.873801 3297 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.875816 1924 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.901108 1746 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:19.908702 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:19.914278 1842 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:19.914894 1842 raft_consensus.cc:604] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership
I20250626 08:23:19.905431 1843 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:19.948552 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:19.949026 1841 raft_consensus.cc:604] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership
W20250626 08:23:19.949992 1733 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 171f89014c824d17a510a3cc17b3af0d already in progress
I20250626 08:23:19.953161 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:19.953738 1843 raft_consensus.cc:604] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership
I20250626 08:23:19.958626 3408 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250626 08:23:19.959501 1795 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 171f89014c824d17a510a3cc17b3af0d already in progress
I20250626 08:23:19.959728 3408 tablet_bootstrap.cc:492] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
W20250626 08:23:19.960029 1795 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 171f89014c824d17a510a3cc17b3af0d already in progress
I20250626 08:23:19.978106 3408 ts_tablet_manager.cc:1397] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.105s user 0.023s sys 0.009s
I20250626 08:23:19.981490 3408 raft_consensus.cc:357] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.982522 3408 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:19.983335 3408 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:19.994153 3408 ts_tablet_manager.cc:1428] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20250626 08:23:20.004016 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cef3ff80065343cdbaf58ef8b1bbedf9 received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:20.004601 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-cef3ff80065343cdbaf58ef8b1bbedf9 on tablet cef3ff80065343cdbaf58ef8b1bbedf9 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:20.070640 3408 ts_tablet_manager.cc:927] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:20.083048 3408 tablet_copy_client.cc:323] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:20.084929 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet a71e0323b96e4a47b1ba2e0a3d72840f from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:20.085450 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet a71e0323b96e4a47b1ba2e0a3d72840f from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-a71e0323b96e4a47b1ba2e0a3d72840f
I20250626 08:23:20.095160 2012 tablet_copy_source_session.cc:215] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:20.116741 3408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a71e0323b96e4a47b1ba2e0a3d72840f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:20.136411 3408 tablet_copy_client.cc:806] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:20.137019 3408 tablet_copy_client.cc:670] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:20.161841 1926 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4f29b6256843489397f6e58f34751764"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:20.163926 1926 raft_consensus.cc:604] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:20.184352 1997 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:20.184983 1997 raft_consensus.cc:604] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:20.188283 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:20.188808 1843 raft_consensus.cc:604] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:20.189282 3408 tablet_copy_client.cc:538] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:20.227947 2585 raft_consensus.cc:1215] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250626 08:23:20.306452 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45556us
W20250626 08:23:20.311942 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 50068.8us
I20250626 08:23:20.297943 1918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:20.320816 3422 ts_tablet_manager.cc:927] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
I20250626 08:23:20.319536 1918 raft_consensus.cc:604] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:20.389415 3408 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
W20250626 08:23:20.458894 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 151568us
W20250626 08:23:20.813936 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.008s sys 0.010s
W20250626 08:23:20.845985 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.539s user 0.009s sys 0.010s
W20250626 08:23:20.846321 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 539049us
W20250626 08:23:20.864166 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.003s sys 0.013s
W20250626 08:23:20.867913 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.003s sys 0.013s
I20250626 08:23:20.871994 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 08:23:20.874148 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:20.874997 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:23:20.887499 1892 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.021s sys 0.002s
W20250626 08:23:20.887838 1892 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.575s user 0.021s sys 0.002s
W20250626 08:23:20.887187 3422 thread.cc:640] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.006s sys 0.026s
W20250626 08:23:20.900828 3422 thread.cc:606] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.516s user 0.006s sys 0.026s
W20250626 08:23:20.918484 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 156300us
I20250626 08:23:20.939090 3422 tablet_copy_client.cc:323] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.193:43311
I20250626 08:23:21.057893 3408 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:21.061941 3408 tablet_bootstrap.cc:492] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:21.062835 3408 ts_tablet_manager.cc:1397] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.674s user 0.031s sys 0.022s
I20250626 08:23:21.066166 3408 raft_consensus.cc:357] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.067655 3408 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:21.068459 3408 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.084997 3426 raft_consensus.cc:991] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:21.085629 3436 raft_consensus.cc:1079] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
W20250626 08:23:21.086632 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 159451us
I20250626 08:23:21.087528 1843 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:21.088295 1843 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:21.088734 1843 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:21.115790 3408 ts_tablet_manager.cc:1428] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250626 08:23:21.117964 1843 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.124135 1843 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:21.124356 2585 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.015s sys 0.039s
W20250626 08:23:21.124691 2585 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.015s sys 0.039s
I20250626 08:23:21.136934 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:21.134090 3448 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:21.141651 3448 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.137589 1926 raft_consensus.cc:3053] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:21.144588 1926 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:21.147204 1926 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.149086 1926 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:21.149791 3420 raft_consensus.cc:991] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:21.150302 3446 raft_consensus.cc:1079] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:21.161691 3448 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.164112 3420 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.166146 1920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:21.167058 1920 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:21.167346 1920 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:21.166630 3426 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.168011 3420 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.168322 3426 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.168707 3436 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.169327 3436 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.170565 3426 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.171407 3436 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.171797 3448 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:21.172295 3448 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.173480 3436 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.174180 3436 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.174520 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.176236 3436 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.177182 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.174187 3448 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.181437 1926 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:21.184262 1814 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:21.184994 3434 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:21.185626 3434 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:21.181886 1920 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.192230 3441 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:21.193463 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.193578 3441 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.196861 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.199213 3443 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:21.199757 3443 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.205453 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:21.208868 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:21.209558 1843 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:21.209960 1843 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:21.192945 1920 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.210904 1843 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.212929 1843 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:21.218921 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:21.224663 1843 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:21.226043 1999 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:21.227383 2585 raft_consensus.cc:1215] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250626 08:23:21.231787 3444 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.232089 3441 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:21.186527 3434 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.229707 1893 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:21.233060 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:21.206017 3420 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.225965 3436 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:21.237726 3426 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:21.238983 3449 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.232447 3444 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.336360 3452 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.337142 1999 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:21.328814 1861 tablet_copy_service.cc:140] P 548099d81e7f46ac9b2f03d9e5e678a6: Received BeginTabletCopySession request for tablet 171f89014c824d17a510a3cc17b3af0d from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41946)
I20250626 08:23:21.343461 3425 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.346731 1861 tablet_copy_service.cc:161] P 548099d81e7f46ac9b2f03d9e5e678a6: Beginning new tablet copy session on tablet 171f89014c824d17a510a3cc17b3af0d from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41946: session id = be56082cacb54e9aa8f1a9ee91e70d76-171f89014c824d17a510a3cc17b3af0d
I20250626 08:23:21.334321 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.347034 3425 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.338621 3446 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:21.358669 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 100739us
I20250626 08:23:21.338577 3420 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.360343 3420 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.360810 3437 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.336925 3452 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.329704 3449 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.441085 3336 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.441743 3336 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.361375 3437 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:21.481822 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:21.483821 3437 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.453004 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:21.455120 3451 ts_tablet_manager.cc:927] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.492494 2001 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:21.510761 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.442332 3452 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.454864 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-a71e0323b96e4a47b1ba2e0a3d72840f received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:21.456923 3420 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.443527 3444 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.556636 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:21.454370 3455 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:21.458824 3441 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.443738 3430 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.454332 3297 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.458279 3435 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.448012 3425 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.443125 3449 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.443835 3336 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.491816 3426 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:21.561479 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.566293 1861 tablet_copy_source_session.cc:215] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:21.569082 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-a71e0323b96e4a47b1ba2e0a3d72840f on tablet a71e0323b96e4a47b1ba2e0a3d72840f with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:21.573351 3440 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.573956 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.574280 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:21.578825 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 214128us
I20250626 08:23:21.454444 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.582659 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.562350 3443 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:21.612775 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.581413 3420 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.614835 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.624740 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.364280 3446 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.580920 3430 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.654143 3430 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.578868 3441 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.579605 3297 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:21.661451 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:21.615214 3452 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:21.657903 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:21.621477 3443 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.721269 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.733398 3423 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:21.734431 3457 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.735019 3441 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.735139 3457 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:21.740217 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 75070.1us
I20250626 08:23:21.626444 3420 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.577189 3455 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.763622 3444 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.764127 3448 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:21.764758 692 debug-util.cc:398] Leaking SignalData structure 0x7b080000ed80 after lost signal to thread 2550
I20250626 08:23:21.765347 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.775761 3445 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.768460 3432 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.778663 3297 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:21.731170 1949 debug-util.cc:398] Leaking SignalData structure 0x7b080012ffc0 after lost signal to thread 690
I20250626 08:23:21.793335 3446 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:21.795691 1949 debug-util.cc:398] Leaking SignalData structure 0x7b0800196820 after lost signal to thread 1781
W20250626 08:23:21.797235 1949 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4240 after lost signal to thread 1875
W20250626 08:23:21.798199 692 kernel_stack_watchdog.cc:198] Thread 2550 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 312ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20250626 08:23:21.800585 3420 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.730535 3443 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.803130 3443 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.723664 3452 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.817188 3438 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.607097 3440 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.817993 3438 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.583741 3435 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.785019 3448 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.821940 3446 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.787851 3444 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.858816 3452 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:21.844218 1949 debug-util.cc:398] Leaking SignalData structure 0x7b080035ab60 after lost signal to thread 1952
I20250626 08:23:21.868500 3435 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.736694 3426 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:21.849327 3455 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:21.819021 3445 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:21.863600 3438 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:21.880610 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 257984us
I20250626 08:23:21.813086 3432 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:21.878616 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 273378us
I20250626 08:23:21.857359 3457 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.882390 2001 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:21.895352 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.896680 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.862668 3446 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.939056 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.939904 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:21.941350 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 159413us
I20250626 08:23:21.948169 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.975252 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.982249 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:21.982434 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.982754 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:21.985062 3425 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:21.985620 3425 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:21.995077 3448 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.997530 3425 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.998591 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.999087 3444 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.000744 3422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171f89014c824d17a510a3cc17b3af0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:22.022085 1816 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.025902 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.028497 3432 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.033208 3398 raft_consensus.cc:991] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:22.036792 3427 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.037515 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.033764 3398 raft_consensus.cc:1079] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:22.033706 3446 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.049619 1890 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.050201 3449 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:22.024194 3451 tablet_copy_client.cc:323] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.195:38293
I20250626 08:23:22.051818 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:22.053308 1949 debug-util.cc:398] Leaking SignalData structure 0x7b0800100e20 after lost signal to thread 2027
I20250626 08:23:22.055222 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.062525 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:21.929759 3428 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:21.934864 3445 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.073071 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.073364 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.074491 2012 tablet_copy_service.cc:140] P 755d83ca0913406b816e95415d502055: Received BeginTabletCopySession request for tablet 70ce63b4769e4ca8980ddb20229389ef from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:41790)
I20250626 08:23:22.075074 2012 tablet_copy_service.cc:161] P 755d83ca0913406b816e95415d502055: Beginning new tablet copy session on tablet 70ce63b4769e4ca8980ddb20229389ef from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:41790: session id = be56082cacb54e9aa8f1a9ee91e70d76-70ce63b4769e4ca8980ddb20229389ef
I20250626 08:23:22.071484 1890 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:22.080691 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.830s user 0.034s sys 0.050s
I20250626 08:23:22.080751 3425 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.066407 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.066612 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:22.098882 1949 debug-util.cc:398] Leaking SignalData structure 0x7b080034bde0 after lost signal to thread 2611
I20250626 08:23:22.058348 3449 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:22.099547 1949 debug-util.cc:398] Leaking SignalData structure 0x7b080033b760 after lost signal to thread 3387
I20250626 08:23:22.066699 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:22.100126 1949 debug-util.cc:398] Leaking SignalData structure 0x7b0800209620 after lost signal to thread 3417
W20250626 08:23:22.100397 1949 debug-util.cc:398] Leaking SignalData structure 0x7b0800300fc0 after lost signal to thread 3418
I20250626 08:23:22.100811 3449 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:22.067287 3332 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.069077 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.102640 1995 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:21.943012 3440 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:21.934296 3336 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:22.104548 3336 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.106518 3336 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:21.939646 3457 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:22.107154 3457 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.109099 3457 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.070258 3428 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.111518 3428 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.112587 2012 tablet_copy_source_session.cc:215] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:22.112915 3456 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:22.117650 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.118481 3456 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.121704 3456 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.125674 3451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ce63b4769e4ca8980ddb20229389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:22.127281 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:22.131660 1892 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.094s sys 0.121s
W20250626 08:23:22.132110 1892 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.094s sys 0.121s
I20250626 08:23:22.132149 3336 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:22.132649 3336 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:22.133114 3336 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:22.134271 3434 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.023s sys 0.013s
W20250626 08:23:22.134649 3434 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.023s sys 0.013s
W20250626 08:23:22.142297 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.892s user 0.034s sys 0.050s
I20250626 08:23:22.146968 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:23:22.214224 1891 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.017s sys 0.023s
W20250626 08:23:22.214836 1891 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.017s sys 0.023s
I20250626 08:23:22.216369 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.217286 1920 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:22.222342 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 81643.1us
I20250626 08:23:22.252597 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.254426 3469 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:22.255124 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.254992 3469 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.257007 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.257119 3469 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:22.258522 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.259676 2582 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4f29b6256843489397f6e58f34751764"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36150
I20250626 08:23:22.260232 2582 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:22.260594 2582 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.252674 1743 catalog_manager.cc:5582] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:22.264863 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.267895 2582 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.269227 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.269322 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.270756 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.272265 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.274307 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.275476 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.276479 3336 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.280212 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.282276 3426 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.284679 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.285104 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.285989 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.289124 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.290750 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.293632 2550 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.294976 3452 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.295488 2550 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.296648 3453 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:22.297863 2550 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:22.299338 3396 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.300899 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.301251 3474 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:22.300961 2550 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:22.301713 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.301765 3474 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.302521 3396 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:22.303085 2550 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.304121 3396 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.307598 2582 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.314940 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.315248 3474 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.308629 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.311138 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.315891 3471 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.311744 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:22.311836 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.312702 3336 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:22.314194 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:22.319289 1996 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.324883 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.325002 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.326328 3458 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.328954 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.330606 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:22.331233 1926 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:22.331560 1926 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:22.332352 1926 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.333348 1926 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.339150 3422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36142 (request call id 406) took 2112 ms. Trace:
I20250626 08:23:22.339632 3422 rpcz_store.cc:276] 0626 08:23:20.226970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:20.234475 (+ 7505us) service_pool.cc:225] Handling call
0626 08:23:20.326171 (+ 91696us) ts_tablet_manager.cc:928] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311)
0626 08:23:21.890775 (+1564604us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:22.076844 (+186069us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:22.077087 (+ 243us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:22.338447 (+261360us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:22.338720 (+ 273us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:22.339010 (+ 290us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:22.339066 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":14746,"tablet-copy.queue_time_us":57265,"thread_start_us":579494,"threads_started":2}
I20250626 08:23:22.341169 3422 tablet_copy_client.cc:806] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:22.341696 3422 tablet_copy_client.cc:670] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:22.342415 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.349534 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.379505 1924 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:22.350183 1814 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.350579 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.358000 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.359295 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.378928 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.381534 1847 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:22.381937 1842 raft_consensus.cc:2391] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:22.362921 3449 raft_consensus.cc:1062] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:22.382254 3474 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.384371 3449 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.363917 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.329314 1996 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:22.368539 1926 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:22.351063 1893 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.387001 3458 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:22.377341 3451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36144 (request call id 328) took 1191 ms. Trace:
I20250626 08:23:22.378644 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.380934 1814 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:22.386262 1812 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.387153 2551 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:22.395817 1814 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.397553 3434 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:22.397879 3453 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:22.390661 1998 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250626 08:23:22.391249 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.397953 3434 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:22.405714 3434 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:22.407544 3466 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.408331 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.389763 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:22.411929 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:22.416393 3466 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.417739 1893 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.402084 3301 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.398517 3453 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:22.402590 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.388379 3458 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.394493 3451 rpcz_store.cc:276] 0626 08:23:21.185809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:21.185967 (+ 158us) service_pool.cc:225] Handling call
0626 08:23:21.536166 (+350199us) ts_tablet_manager.cc:928] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
0626 08:23:22.124155 (+587989us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:22.311926 (+187771us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:22.312160 (+ 234us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:22.376551 (+ 64391us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:22.376861 (+ 310us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:22.377197 (+ 336us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:22.377257 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":68316,"tablet-copy.queue_time_us":63450,"thread_start_us":317636,"threads_started":2}
I20250626 08:23:22.403867 3463 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.395285 1741 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:22.401360 1812 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.423012 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.426906 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.427745 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.419814 3434 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.424840 3451 tablet_copy_client.cc:806] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:22.436594 1849 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:22.431741 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.432250 3458 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.434054 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.426270 3466 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.428555 3336 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.428511 3262 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.428525 3301 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.447666 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.436048 3429 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250626 08:23:22.449033 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.442647 3451 tablet_copy_client.cc:670] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:22.429208 1814 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.433008 3474 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.444640 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.452766 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:22.454197 3467 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.451896 3469 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.457966 3463 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.424597 3453 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.459921 1814 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.461656 3463 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.463873 3442 raft_consensus.cc:1062] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:22.465941 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:22.468634 3451 tablet_copy_client.cc:538] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:22.468773 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:22.468971 1923 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:22.471081 1814 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.472394 2584 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:22.474522 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:22.474591 3467 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:22.476781 3469 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:22.485358 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:22.487702 3434 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.496826 3452 consensus_queue.cc:1046] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:22.498364 3396 raft_consensus.cc:3053] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:22.498906 3396 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:22.499884 3396 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.501926 3301 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.501658 3396 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.506160 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.505678 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:22.466950 3442 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:22.524832 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.530143 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.535333 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:22.526944 3466 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.550383 1812 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.554641 1742 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:22.532840 3439 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.534346 3422 tablet_copy_client.cc:538] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:22.531850 3430 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.580595 1812 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.584818 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:22.592025 1996 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:22.593870 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:22.594807 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 84640.7us
I20250626 08:23:22.594861 1999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8353d62d68914958b13bb87541c357f8"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36848
W20250626 08:23:22.595522 1890 consensus_peers.cc:487] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:22.595408 1968 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:22.581801 3301 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:22.583443 3283 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.598609 1968 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:22.595395 1999 raft_consensus.cc:604] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Received request to transfer leadership to TS be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.604773 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.605732 1968 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:22.609299 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:22.596982 3469 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.606230 3467 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.597658 3301 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:22.614161 3301 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:22.606475 3449 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.603577 3297 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.606631 3428 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.644523 1814 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.647178 1847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dc905d51bc4a450484091ffa39379396"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:22.647831 1847 raft_consensus.cc:604] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:22.648223 3445 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:22.650050 1842 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 08:23:22.655718 3336 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:22.617631 3476 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.635102 3469 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.672396 1921 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:22.674242 1921 raft_consensus.cc:604] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:22.678404 3467 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:22.663784 3451 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:22.684252 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:22.685982 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:22.686717 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:22.731081 1921 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:22.768512 2580 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:22.778092 3492 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:22.780323 3301 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.781724 3425 raft_consensus.cc:991] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:22.782259 3425 raft_consensus.cc:1079] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:22.782779 3301 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.787276 1999 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:22.789119 1850 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:22.789065 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:22.790503 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:22.745721 3446 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20250626 08:23:22.792018 3422 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:22.801754 3474 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:22.765391 1919 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:22.811307 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 70226.4us
I20250626 08:23:22.816257 1849 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250626 08:23:22.817057 1847 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:22.817483 1847 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:22.817693 1847 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:22.828593 3449 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:22.854070 3493 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:22.861706 1847 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.868341 1847 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:22.870066 3283 raft_consensus.cc:991] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:22.870630 3283 raft_consensus.cc:1079] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:22.809258 3336 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20250626 08:23:22.873657 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:22.874462 2002 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:22.874660 1996 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dc905d51bc4a450484091ffa39379396"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:22.875406 1996 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:22.875726 1996 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.875638 1841 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:22.896111 2002 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:22.908025 1816 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:22.910652 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:22.913138 1996 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
W20250626 08:23:22.917200 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 125359us
I20250626 08:23:22.919638 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:22.920212 1923 raft_consensus.cc:3053] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:22.920491 1923 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:22.921293 1923 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.922358 3438 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:22.923403 3301 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:22.923940 3301 raft_consensus.cc:695] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:22.922600 1923 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:22.924825 3301 consensus_queue.cc:237] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.932400 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:22.933041 1921 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:22.931509 1996 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:22.940351 1921 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:22.941869 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:22.942870 3443 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:22.943675 3443 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:22.944567 3443 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:22.951521 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:22.953115 1842 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:22.955811 1923 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:22.953498 1842 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:22.966845 3449 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:22.968298 3449 raft_consensus.cc:3053] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:22.968825 3449 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:22.969930 3449 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:22.971863 3449 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:22.975994 3467 raft_consensus.cc:991] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: : Instructing follower be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:22.976727 3467 raft_consensus.cc:1079] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Signalling peer be56082cacb54e9aa8f1a9ee91e70d76 to start an election
I20250626 08:23:22.975930 2582 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:22.984316 2583 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8353d62d68914958b13bb87541c357f8"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
from {username='slave'} at 127.0.0.1:36144
I20250626 08:23:22.985033 2583 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:22.985410 2583 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:23.012821 1743 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 4, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.024744 2583 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.028254 1998 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:23.030709 3396 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250626 08:23:22.962761 1842 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:23.063091 1841 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:23.051332 1842 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:23.081984 3467 consensus_queue.cc:1046] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250626 08:23:23.084100 2583 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:23.084419 3467 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:23.084995 3467 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:23.082794 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:23.092506 1921 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:23.087416 1841 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:23.099335 3453 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:23.094859 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:23.103400 1998 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:23.103776 1998 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:23.112154 1998 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.113407 1998 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:23.103495 3467 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:23.137270 3467 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:23.131489 1842 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:23.143723 1841 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:23.150388 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:23.183187 1742 catalog_manager.cc:5582] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:23.157629 3453 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:23.207152 1998 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:23.225862 3440 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.243986 1746 catalog_manager.cc:5582] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:23.242825 1923 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.244902 2585 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.262696 1996 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.231417 3453 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:23.267109 3453 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.270175 2583 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:23.278936 3467 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:23.296257 1923 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:23.297545 3443 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:23.300928 1742 catalog_manager.cc:5582] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.309505 1919 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:23.312289 3262 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:23.318965 3449 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:23.302439 3422 log.cc:927] Time spent T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.074s user 0.002s sys 0.001s
I20250626 08:23:23.334803 2583 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.343727 3451 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:23.344935 3451 tablet_bootstrap.cc:492] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:23.345691 3451 ts_tablet_manager.cc:1397] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.724s user 0.033s sys 0.009s
I20250626 08:23:23.350651 1842 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:23.357700 1923 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:23.371572 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a71e0323b96e4a47b1ba2e0a3d72840f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:23.362195 3467 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250626 08:23:23.382220 1923 raft_consensus.cc:2953] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.383190 3443 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:23.391461 1742 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 4 to 5, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:23.395370 3422 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:23.414713 1743 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:23.416330 1996 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:23.392182 3451 raft_consensus.cc:357] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:23.418494 3262 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250626 08:23:23.418018 3451 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:23.426811 3422 tablet_bootstrap.cc:492] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:23.459026 3422 ts_tablet_manager.cc:1397] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 0.701s user 0.020s sys 0.023s
I20250626 08:23:23.460253 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet a71e0323b96e4a47b1ba2e0a3d72840f with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:23.483222 1998 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.459393 3451 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:23.515198 3422 raft_consensus.cc:357] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:23.516062 3422 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:23.516680 3422 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:23.550055 3451 ts_tablet_manager.cc:1428] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.204s user 0.009s sys 0.000s
I20250626 08:23:23.552259 2012 tablet_copy_service.cc:342] P 755d83ca0913406b816e95415d502055: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-70ce63b4769e4ca8980ddb20229389ef received from {username='slave'} at 127.0.0.1:41790
I20250626 08:23:23.557492 2012 tablet_copy_service.cc:434] P 755d83ca0913406b816e95415d502055: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-70ce63b4769e4ca8980ddb20229389ef on tablet 70ce63b4769e4ca8980ddb20229389ef with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:23.558238 3422 ts_tablet_manager.cc:1428] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 0.093s user 0.003s sys 0.003s
I20250626 08:23:23.563692 1861 tablet_copy_service.cc:342] P 548099d81e7f46ac9b2f03d9e5e678a6: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-171f89014c824d17a510a3cc17b3af0d received from {username='slave'} at 127.0.0.1:41946
I20250626 08:23:23.564870 1861 tablet_copy_service.cc:434] P 548099d81e7f46ac9b2f03d9e5e678a6: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-171f89014c824d17a510a3cc17b3af0d on tablet 171f89014c824d17a510a3cc17b3af0d with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:23.585666 3519 tablet_replica.cc:331] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:23.586726 3519 raft_consensus.cc:2241] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:23.587482 3519 raft_consensus.cc:2270] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:23.599344 1843 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.632151 3519 ts_tablet_manager.cc:1905] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:23.632167 2585 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:23.639881 2002 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:23.645342 3446 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:23.652189 3301 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.663861 2002 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:23.664338 1999 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.665438 3453 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:23.705304 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cef3ff80065343cdbaf58ef8b1bbedf9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:23.712599 3445 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:23.713313 1742 catalog_manager.cc:5582] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:23.727058 2585 raft_consensus.cc:2953] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:23.729966 1842 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:23.743568 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet cef3ff80065343cdbaf58ef8b1bbedf9 with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:23.743286 3452 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:23.771047 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:23.791875 3519 ts_tablet_manager.cc:1918] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250626 08:23:23.792399 3519 log.cc:1199] T a71e0323b96e4a47b1ba2e0a3d72840f P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/a71e0323b96e4a47b1ba2e0a3d72840f
I20250626 08:23:23.794586 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet a71e0323b96e4a47b1ba2e0a3d72840f (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:23.796216 3522 tablet_replica.cc:331] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:23.797513 3522 raft_consensus.cc:2241] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:23.798383 3522 raft_consensus.cc:2270] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:23.810388 3513 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:23.810914 3513 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.812896 3513 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:23.815330 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:23.816130 1999 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:23.822539 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:23.823442 2583 raft_consensus.cc:1215] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 08:23:23.824208 3522 ts_tablet_manager.cc:1905] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:23.826217 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:23.826915 1842 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:23.823490 3513 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:23.839710 3513 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:23.840062 3513 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:23.895459 3513 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.898196 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:23.898842 1842 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:23.899359 3513 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:23.902253 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:23.902860 1999 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:23.909046 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:23.909638 1843 raft_consensus.cc:604] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:23.921296 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3946d60648894026984085c3abee51ea"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:23.923163 1843 raft_consensus.cc:604] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:23.928236 1998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:23.928833 1998 raft_consensus.cc:604] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:23.931842 1999 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:23.933025 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:23.934599 3513 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:23.934944 1842 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:23.935282 3513 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:23.935950 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:23.937188 3513 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:23.938115 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:23.938905 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:23.934293 3522 ts_tablet_manager.cc:1918] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:23.950160 3522 log.cc:1199] T cef3ff80065343cdbaf58ef8b1bbedf9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/cef3ff80065343cdbaf58ef8b1bbedf9
I20250626 08:23:23.952358 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet cef3ff80065343cdbaf58ef8b1bbedf9 (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:23.985741 1742 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:24.001581 3445 raft_consensus.cc:991] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:24.002070 3445 raft_consensus.cc:1079] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:24.004704 1921 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3946d60648894026984085c3abee51ea"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:24.005234 1921 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:24.005568 1921 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.013718 3457 raft_consensus.cc:1062] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:24.016619 1921 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.017675 3457 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.021243 1921 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.022249 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:24.022747 2002 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.026131 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:24.026777 1842 raft_consensus.cc:3053] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:24.027124 1842 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:24.027901 1842 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.029142 1842 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.037953 1842 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:24.044965 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:24.045637 3439 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:24.046170 3439 raft_consensus.cc:695] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:24.046973 3439 consensus_queue.cc:237] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.048260 1998 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:24.049778 2582 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:24.050770 3332 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:24.055732 3458 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:24.064945 1843 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:24.067783 3458 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:24.083122 1742 catalog_manager.cc:5582] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:24.090261 2002 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:24.107472 3441 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:24.108000 3441 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.109961 3441 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:24.115149 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:24.115875 1843 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:24.116631 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:24.117259 1968 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:24.122001 3441 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:24.122431 3441 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:24.122783 3441 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:24.142005 3458 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.143858 3441 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.146821 3441 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:24.147040 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:24.146729 2583 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.161370 1850 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.165067 2002 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.172458 1742 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 3 to 5, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:24.179620 3301 raft_consensus.cc:991] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:24.180205 3301 raft_consensus.cc:1079] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:24.190639 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:24.194799 1919 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:24.195508 1919 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:24.195860 1919 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.211473 1923 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.229656 2583 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:24.231320 1843 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:24.240834 1919 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.231420 2001 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:24.242710 1919 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.243965 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:24.244531 1996 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.235270 3441 raft_consensus.cc:991] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:24.250751 3469 raft_consensus.cc:1079] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:24.250928 3457 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:24.252951 1996 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:24.254137 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:24.255863 3457 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:24.256318 3457 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:24.257169 3457 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.267244 3527 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:24.267822 3527 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.269899 3527 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.271668 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:24.272622 1999 raft_consensus.cc:2366] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:23:24.271616 3332 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.273876 1893 leader_election.cc:400] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:23:24.274199 1893 leader_election.cc:403] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:23:24.275281 3528 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 3
I20250626 08:23:24.275205 2002 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.279547 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:24.280191 1842 raft_consensus.cc:3053] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:24.280500 1842 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:24.287034 3527 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:24.298610 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:24.311663 1850 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:24.315135 1850 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:24.315498 1850 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.310978 1842 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.323365 1842 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.326366 1850 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.329219 1850 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:24.340147 1732 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 171f89014c824d17a510a3cc17b3af0d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:24.346051 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:24.345604 1742 catalog_manager.cc:5582] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:24.346674 1998 raft_consensus.cc:3053] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:24.347054 1998 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:24.348089 3446 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:24.348606 3446 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.350996 3446 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.350418 1743 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 reported cstate change: config changed from index 5 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.351655 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:24.347779 1998 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.352947 1998 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.354610 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:24.355237 2002 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:24.356232 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:24.357628 3446 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:24.358031 3446 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:24.358559 3446 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:24.362000 2583 raft_consensus.cc:2953] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:24.364639 3446 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:24.358541 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 45972.2us
I20250626 08:23:24.368081 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:24.368685 2582 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:24.367476 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:24.374634 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:24.375598 3446 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.375598 2582 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:24.384202 1843 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:24.387887 1998 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:24.375209 2002 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:24.389052 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:24.391253 3434 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:24.391408 1836 tablet_service.cc:1515] Processing DeleteTablet for tablet 171f89014c824d17a510a3cc17b3af0d with delete_type TABLET_DATA_TOMBSTONED (TS 548099d81e7f46ac9b2f03d9e5e678a6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60106
I20250626 08:23:24.391816 3434 raft_consensus.cc:695] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:24.392588 3528 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:23:24.392755 3434 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.403664 3332 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:24.404220 3332 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.390455 1968 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:24.406424 3332 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:24.406473 3469 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:24.407153 3469 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:24.407999 3469 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.408365 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:24.416775 1847 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:24.417883 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:24.418666 1843 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:24.424082 3441 raft_consensus.cc:1062] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:24.427227 3441 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:24.429509 3513 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:24.432674 1893 leader_election.cc:400] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:24.426580 1842 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:24.432984 1893 leader_election.cc:403] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:23:24.434866 3458 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 5
I20250626 08:23:24.459656 1843 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:24.460601 3532 tablet_replica.cc:331] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: stopping tablet replica
I20250626 08:23:24.461941 3532 raft_consensus.cc:2241] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:24.463356 3532 raft_consensus.cc:2270] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:24.471156 2002 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:24.472868 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:24.474124 3301 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:24.482056 1743 catalog_manager.cc:5582] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:24.486541 3532 ts_tablet_manager.cc:1905] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:24.466665 3439 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20250626 08:23:24.491741 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45069.4us
I20250626 08:23:24.493881 1742 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:24.509423 3301 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:24.521106 3301 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:24.523221 1996 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:24.525435 3458 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:24.528048 3457 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:24.532246 2002 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:24.534667 3513 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20250626 08:23:24.541260 1923 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:24.568769 3441 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20250626 08:23:24.590646 1850 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:24.592798 3492 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:24.594522 2582 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 08:23:24.596865 3492 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:24.671387 3532 ts_tablet_manager.cc:1918] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250626 08:23:24.672174 3532 log.cc:1199] T 171f89014c824d17a510a3cc17b3af0d P 548099d81e7f46ac9b2f03d9e5e678a6: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-0-root/wals/171f89014c824d17a510a3cc17b3af0d
I20250626 08:23:24.705819 1850 raft_consensus.cc:1273] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:24.710151 3557 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250626 08:23:24.918901 1999 raft_consensus.cc:1273] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:24.922374 1733 catalog_manager.cc:4928] TS 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): tablet 171f89014c824d17a510a3cc17b3af0d (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
W20250626 08:23:24.914669 3545 log.cc:927] Time spent T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250626 08:23:24.952061 2585 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:24.988091 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 83308.7us
I20250626 08:23:24.993366 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:25.008852 2001 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:25.020639 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 89583.3us
W20250626 08:23:25.130033 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 103479us
W20250626 08:23:25.137426 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 213814us
W20250626 08:23:25.180202 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 87820.4us
I20250626 08:23:25.087960 3581 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
W20250626 08:23:25.215750 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 122761us
I20250626 08:23:25.339058 1849 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250626 08:23:25.344147 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 130932us
I20250626 08:23:25.389140 3538 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:25.378225 3469 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
W20250626 08:23:25.395802 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 46634us
W20250626 08:23:25.415552 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 391939us
I20250626 08:23:25.421388 1919 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:25.420432 3547 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:25.438023 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 91550.7us
I20250626 08:23:25.444633 3577 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20250626 08:23:25.515841 1919 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:25.546259 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 202376us
W20250626 08:23:25.554622 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 108276us
I20250626 08:23:25.585742 2001 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:23:25.634145 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 130456us
I20250626 08:23:25.563656 3469 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250626 08:23:25.650739 1919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:25.651580 1919 raft_consensus.cc:604] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:25.654352 3582 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.361s user 0.002s sys 0.014s
I20250626 08:23:25.666270 1843 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:25.680989 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 93438.7us
I20250626 08:23:25.684283 1919 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:23:25.690891 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 135052us
I20250626 08:23:25.729444 2582 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:25.730377 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 89131.6us
I20250626 08:23:25.733743 3602 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:25.751647 1843 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:25.752377 1843 raft_consensus.cc:604] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
W20250626 08:23:25.785391 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 197086us
I20250626 08:23:25.753161 3602 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20250626 08:23:25.796778 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45589.4us
W20250626 08:23:25.815837 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 64483.8us
I20250626 08:23:25.824873 1919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:25.825464 1919 raft_consensus.cc:604] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:25.842682 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:25.845093 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:25.846899 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:25.936177 1746 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index 2 to 3, be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } attrs { replace: true } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:26.012915 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 49765.8us
I20250626 08:23:26.062433 3568 raft_consensus.cc:991] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:26.063000 3439 raft_consensus.cc:1079] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
W20250626 08:23:26.016469 3614 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250626 08:23:26.071803 1993 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34656
I20250626 08:23:26.072413 1993 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:26.072806 1993 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:26.105572 1742 catalog_manager.cc:5582] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 3 to 5, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 5 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:26.131760 3599 raft_consensus.cc:991] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:26.132458 3457 raft_consensus.cc:1079] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:26.149257 1849 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:26.149892 1849 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:26.150290 1849 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.159132 1993 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.163007 1993 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:26.175954 3579 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:26.177706 3579 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.180373 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:26.180984 1841 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:26.185526 3579 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:26.187393 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:26.190834 1849 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.197415 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:26.208626 1925 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:26.208897 1843 raft_consensus.cc:1238] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:26.208982 1925 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:26.209830 1925 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.210355 1849 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:26.211073 1925 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:26.215262 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:26.218745 1841 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:26.221078 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:26.221958 3579 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:26.223551 3562 raft_consensus.cc:991] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:26.223903 1968 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:26.225317 3573 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:26.226048 3573 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:26.227301 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:26.227540 3579 consensus_queue.cc:1046] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:26.228003 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:26.228533 1923 raft_consensus.cc:3053] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:26.228819 1923 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:26.229535 1923 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.230911 3562 raft_consensus.cc:1079] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:26.235219 1923 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.226927 3573 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.242866 1923 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:26.244469 1999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:26.245126 1999 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:26.246004 1999 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.250604 1925 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:26.255542 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:26.257735 3562 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:26.258311 3562 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:26.259210 3562 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.312491 1999 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.316234 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:26.317487 1849 raft_consensus.cc:3053] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:26.317876 1849 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:26.318780 1849 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.316234 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:26.320238 1849 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.320268 1921 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.315232 1999 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:26.357815 1921 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:26.360586 1742 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32850 (request call id 292) took 1038 ms. Trace:
I20250626 08:23:26.407752 1742 rpcz_store.cc:276] 0626 08:23:25.321718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:25.321872 (+ 154us) service_pool.cc:225] Handling call
0626 08:23:26.360525 (+1038653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:26.245814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:26.246397 (+ 583us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:26.246436 (+ 39us) write_op.cc:435] Acquired schema lock
0626 08:23:26.246451 (+ 15us) tablet.cc:660] Decoding operations
0626 08:23:26.246793 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:26.246839 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:26.246863 (+ 24us) tablet.cc:683] Acquiring locks for 1 operations
0626 08:23:26.247029 (+ 166us) tablet.cc:699] Row locks acquired
0626 08:23:26.247046 (+ 17us) write_op.cc:260] PREPARE: finished
0626 08:23:26.247182 (+ 136us) write_op.cc:270] Start()
0626 08:23:26.247284 (+ 102us) write_op.cc:276] Timestamp: P: 1750926206247141 usec, L: 0
0626 08:23:26.247303 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 08:23:26.247869 (+ 566us) log.cc:844] Serialized 541 byte log entry
0626 08:23:26.305413 (+ 57544us) op_driver.cc:464] REPLICATION: finished
0626 08:23:26.335885 (+ 30472us) write_op.cc:301] APPLY: starting
0626 08:23:26.335956 (+ 71us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:26.336051 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:26.336148 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:26.341193 (+ 5045us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:26.357964 (+ 16771us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 08:23:26.358024 (+ 60us) write_op.cc:312] APPLY: finished
0626 08:23:26.358223 (+ 199us) log.cc:844] Serialized 23 byte log entry
0626 08:23:26.358388 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:26.358469 (+ 81us) write_op.cc:454] Released schema lock
0626 08:23:26.358505 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0626 08:23:26.358837 (+ 332us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:26.358871 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":362128,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":30292,"apply.run_cpu_time_us":1546,"apply.run_wall_time_us":22912,"num_ops":1,"prepare.queue_time_us":137348,"prepare.run_cpu_time_us":2290,"prepare.run_wall_time_us":2290,"replication_time_us":58000,"thread_start_us":41866,"threads_started":1}]]}
I20250626 08:23:26.373699 1849 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:26.397799 1998 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:26.413440 1968 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:26.414356 3544 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:26.414881 3544 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:26.415772 3544 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:26.413226 1741 catalog_manager.cc:5582] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:26.419034 2585 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:26.420464 3608 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:26.427933 3441 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:26.434290 2585 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:26.445581 1849 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 08:23:26.451072 1733 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 70ce63b4769e4ca8980ddb20229389ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 08:23:26.454463 3544 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:26.458411 1741 catalog_manager.cc:5582] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:26.460129 1849 raft_consensus.cc:2953] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:26.461604 1742 catalog_manager.cc:5582] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:26.488224 1746 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: config changed from index 3 to 4, VOTER 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) evicted. New cstate: current_term: 1 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:26.508235 1914 tablet_service.cc:1515] Processing DeleteTablet for tablet 70ce63b4769e4ca8980ddb20229389ef with delete_type TABLET_DATA_TOMBSTONED (TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39250
I20250626 08:23:26.519186 3630 tablet_replica.cc:331] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: stopping tablet replica
I20250626 08:23:26.520459 3630 raft_consensus.cc:2241] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 08:23:26.521315 3630 raft_consensus.cc:2270] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 08:23:26.534312 3630 ts_tablet_manager.cc:1905] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 08:23:26.554273 3630 ts_tablet_manager.cc:1918] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 08:23:26.554847 3630 log.cc:1199] T 70ce63b4769e4ca8980ddb20229389ef P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Deleting WAL directory at /tmp/dist-test-taskHkgSwb/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1750926095312382-687-0/minicluster-data/ts-1-root/wals/70ce63b4769e4ca8980ddb20229389ef
I20250626 08:23:26.556846 1732 catalog_manager.cc:4928] TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): tablet 70ce63b4769e4ca8980ddb20229389ef (table test-workload [id=f0c35af97a4e48e29b9764c1db09ff04]) successfully deleted
I20250626 08:23:26.721531 1841 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:26.723371 3573 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:26.762753 2001 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:26.772137 2001 raft_consensus.cc:1273] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:26.773707 3562 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:26.822144 1925 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:26.827097 3573 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:26.870669 1923 raft_consensus.cc:1273] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:26.872185 3591 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:26.966506 1841 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:27.006696 3573 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:27.019117 1925 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:27.026535 3441 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:23:27.173777 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 53756.3us
I20250626 08:23:27.354427 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:27.432206 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:27.433156 1841 raft_consensus.cc:604] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:27.441154 2001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:27.441845 2001 raft_consensus.cc:604] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:27.460673 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:27.461297 1841 raft_consensus.cc:604] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:27.462944 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:27.464663 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:27.465706 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:27.658555 1923 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:27.675908 1890 consensus_peers.cc:487] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac6c556e015045ac949f4ee3d2048b56. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:27.677268 1998 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:27.680092 3599 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:27.680768 1841 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 08:23:27.684998 3457 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:27.697780 3566 raft_consensus.cc:991] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:27.696709 3457 raft_consensus.cc:2953] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:27.702328 1999 raft_consensus.cc:2953] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:27.713510 3573 raft_consensus.cc:991] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:27.714483 3573 raft_consensus.cc:1079] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:27.724795 3591 raft_consensus.cc:1079] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Signalling peer 755d83ca0913406b816e95415d502055 to start an election
I20250626 08:23:27.736217 1998 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a"
dest_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:34646
I20250626 08:23:27.736817 1998 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:27.737192 1998 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:27.740243 1849 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60136
I20250626 08:23:27.740806 1849 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:27.741159 1849 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.749392 1849 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.756614 1998 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:27.759294 1795 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20250626 08:23:27.759596 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:27.760238 2001 raft_consensus.cc:3053] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:27.760516 2001 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:27.761278 2001 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.758679 1849 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:27.762612 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:27.764487 1923 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.764725 2001 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.772593 1923 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:27.774812 1814 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:27.775802 3566 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:27.776938 3566 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:27.778029 3566 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.784236 2001 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:27.795150 1998 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:27.796408 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:27.797024 1925 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:27.807168 1795 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5549f2d4a32b4170ad59fc7489d1365f, TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:27.807634 1795 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5549f2d4a32b4170ad59fc7489d1365f, TS 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.810328 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:27.810930 1850 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:27.811228 1850 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:27.803977 1925 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:27.811902 1850 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.812500 1967 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:27.813014 1850 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:27.813623 3573 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:27.808738 1746 catalog_manager.cc:5582] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New cstate: current_term: 1 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:27.814271 3573 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:27.815200 3573 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.836984 1742 catalog_manager.cc:5582] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:27.876368 1850 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:27.878458 1795 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8e262c3543404a9cb840169098735ea2, TS 755d83ca0913406b816e95415d502055 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:27.878836 1795 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8e262c3543404a9cb840169098735ea2, TS 755d83ca0913406b816e95415d502055 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:27.886638 1795 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 187f08b11130495ba8e2a482370447f4, TS 548099d81e7f46ac9b2f03d9e5e678a6 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:27.886979 1795 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 187f08b11130495ba8e2a482370447f4, TS 548099d81e7f46ac9b2f03d9e5e678a6 -> TS be56082cacb54e9aa8f1a9ee91e70d76 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.921231 3562 raft_consensus.cc:991] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: : Instructing follower 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:27.922056 3566 raft_consensus.cc:1079] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Signalling peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 to start an election
I20250626 08:23:27.924037 1921 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:39254
I20250626 08:23:27.924563 1921 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:27.924901 1921 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.935356 1921 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.944898 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:27.945447 2001 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.947247 1921 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:27.946565 1841 raft_consensus.cc:2953] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:27.954210 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:27.954950 1850 raft_consensus.cc:3053] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:27.962177 1850 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:27.975442 1850 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.976871 1850 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:27.961755 2001 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:27.983068 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:27.983862 3457 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:27.984304 3457 raft_consensus.cc:695] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:27.985204 3457 consensus_queue.cc:237] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:27.998852 1850 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:28.026474 1742 catalog_manager.cc:5582] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:28.024945 1741 catalog_manager.cc:5582] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 3 to 4, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 4 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:23:28.569589 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:23:29.170071 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.010s sys 0.005s
W20250626 08:23:29.170532 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.010s sys 0.005s
I20250626 08:23:29.268003 3662 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.268814 3662 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.272110 3662 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.282269 3662 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.282971 3662 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.286134 3662 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.286667 3662 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.287281 3662 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.309487 3663 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.310364 3663 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.313035 3663 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.370775 3566 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.371583 3566 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.517894 3566 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.518795 3566 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.519483 3566 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.522715 3566 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.590978 3663 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.742615 3566 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.742945 3667 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.743216 3566 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.743762 3667 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.746016 3667 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.746608 3667 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:29.747164 3667 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.749337 3667 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.750013 3667 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.750483 3667 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:29.752259 3667 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:29.706739 3665 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.754070 3665 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.743002 3663 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.757182 3665 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.757177 3663 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.646414 3662 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.738822 3666 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.758666 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 278) took 1445 ms. Trace:
I20250626 08:23:29.759184 3666 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.759135 1847 rpcz_store.cc:276] 0626 08:23:28.313035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:29.757534 (+1444499us) service_pool.cc:225] Handling call
0626 08:23:29.757926 (+ 392us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:29.758144 (+ 218us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:29.758189 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:29.758245 (+ 56us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:29.758289 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:29.758359 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:29.758511 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:29.755477 3668 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.745568 3566 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.760783 3668 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.761250 3664 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.761826 3664 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.763942 3664 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.764403 3667 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.764577 3664 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.764953 3667 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.765167 3664 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:29.767653 3667 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.767755 3664 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.768275 3667 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.768429 3664 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.768849 3667 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.769196 3664 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.770870 3667 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.771235 3664 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.771391 3663 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.771804 3664 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.771931 3663 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.772348 3664 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.772928 1921 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:29.773651 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1615) took 1527 ms. Trace:
I20250626 08:23:29.774200 3662 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.775030 3662 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.778668 3579 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.779369 3579 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.780959 3663 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.781566 3579 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.774077 1921 rpcz_store.cc:276] 0626 08:23:28.245980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:29.772274 (+1526294us) service_pool.cc:225] Handling call
0626 08:23:29.772590 (+ 316us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:29.773328 (+ 738us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:29.773585 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:29.784138 3667 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.784739 3667 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.787794 3560 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.788455 3560 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.799732 3668 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.800976 3667 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.801575 3666 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.802476 3670 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.804037 3453 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.804075 3579 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.791457 3560 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.806090 3453 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.815683 3453 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.816424 3579 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.818665 3579 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.811378 3669 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.819186 3579 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.819661 3579 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.819674 3669 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.825292 3669 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:29.826236 3579 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.826831 3670 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.829437 3670 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.832957 3669 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.833392 3453 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.833524 3669 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.834321 3453 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.836505 3669 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:29.837005 3453 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.847358 3664 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.849238 3674 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.849941 3674 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.852154 3674 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.853174 3673 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.853777 3670 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.853744 3673 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.854393 3670 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.855968 3673 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.856349 3670 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.856482 3673 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.857290 3669 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.857498 3579 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.858173 3671 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.858608 3560 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.859131 3560 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.859823 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.860409 2586 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:29.860922 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:29.861083 3560 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:29.862056 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.862141 3670 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.858639 3671 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.862752 3670 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.863402 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.864509 3671 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.864728 3670 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.865311 3670 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.857820 3669 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.865823 3670 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.867702 3669 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.867851 3670 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.861624 3560 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.868917 3579 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.864724 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.871104 2580 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:29.870223 3560 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.874084 3676 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.874722 3676 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.876829 3676 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.857108 3673 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.877890 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:29.879612 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.880239 2583 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:29.882202 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.883630 3560 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.884212 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:29.884799 3664 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:29.884979 3673 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.885171 3664 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.885349 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:29.885528 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1621) took 1637 ms. Trace:
I20250626 08:23:29.885636 3664 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:29.887207 3579 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.889406 3579 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.889902 3579 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.890560 3666 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:29.890990 3666 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.891328 3666 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:29.892125 3579 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.893668 3671 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.886665 1923 rpcz_store.cc:276] 0626 08:23:28.248111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:28.248298 (+ 187us) service_pool.cc:225] Handling call
0626 08:23:28.248670 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:28.248908 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:28.248945 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:28.248991 (+ 46us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:28.310838 (+ 61847us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:29.885336 (+1574498us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:29.885440 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:29.894201 3671 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.896180 3671 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.896763 3671 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:29.897172 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.897889 2578 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:29.898066 3579 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.899211 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:29.899871 1850 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:29.899736 3579 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.902280 3579 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.906332 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.897311 3671 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.907799 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.908785 3671 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.898294 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:29.905629 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.934679 2585 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:29.902330 3662 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.910380 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.922207 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.898193 3673 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:29.903035 3674 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.942893 3666 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.940871 3673 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.950661 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.952499 3673 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.952589 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:29.953599 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 61278.9us
I20250626 08:23:29.954015 1921 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:29.955519 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.956627 1923 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:29.957804 3677 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.958432 3677 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.960639 3677 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.949436 3666 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.962596 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.965626 3678 ts_tablet_manager.cc:927] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:29.968531 3677 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.954787 3674 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.969980 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.972585 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.967768 3671 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:29.988902 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.986866 3675 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.992144 3683 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:29.992715 3683 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:29.993120 3674 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:29.994033 1923 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:29.994828 3683 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:29.994910 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:29.995579 1919 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:29.997784 3680 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:29.998340 3680 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.000288 3680 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.001766 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.002925 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:29.990127 3682 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.004980 2550 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.005045 3682 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.006155 3672 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.006518 3672 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.006878 3672 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.007144 3682 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.007735 3675 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.978775 3677 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.009802 3675 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.029225 2550 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.014366 3673 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.013815 3683 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.040256 3683 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.043140 3683 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.013562 3682 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.043709 3691 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.044138 3682 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:29.986070 3671 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.044371 3691 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.050812 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.051654 3685 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.051721 2584 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:30.052310 3685 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.056907 3670 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.057420 3670 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.057838 3670 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.059242 3685 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.060043 3685 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.060783 3685 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.062186 3680 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.063020 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.039621 3673 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:30.062959 3680 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.065172 3685 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.069183 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.070171 1928 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:30.070461 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.071362 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.071818 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.072706 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.073796 3677 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.074606 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.075744 3673 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.076248 1928 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:30.082410 3689 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.083299 3689 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.083945 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.084611 1920 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:30.087054 1925 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:30.092970 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.094367 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:30.094606 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 196392us
I20250626 08:23:30.095160 3664 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.104027 1812 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.109203 3677 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:30.109535 3678 tablet_copy_client.cc:323] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.171.194:39813
I20250626 08:23:30.111630 3674 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.112455 3674 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.115128 3667 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:30.115734 3667 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.116228 3667 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:30.118095 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 62344.4us
W20250626 08:23:30.120288 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 136116us
W20250626 08:23:30.121793 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 92702.1us
I20250626 08:23:30.126355 1890 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.129271 2550 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.118808 3665 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.118348 3677 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.120175 3663 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.143718 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:30.144906 2582 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.122561 3566 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.147212 3566 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.147964 3686 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.148506 3686 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.149344 3566 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.150573 3686 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.151067 3690 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.151569 3690 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.154973 3664 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.155392 3674 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.156720 3691 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.158318 3663 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.159188 2582 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:30.160331 3663 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.161077 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.127846 3682 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.131901 3560 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.132572 3668 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.167253 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.167116 3665 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.168120 3664 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.167779 3681 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:30.134198 3680 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.171857 1927 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:30.145117 3662 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.131680 3670 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.162011 3690 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.171542 3668 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.171152 3560 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.171648 3671 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.220629 3560 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.224537 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.218006 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.225226 1924 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:30.219148 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.231565 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:30.238420 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 102560us
I20250626 08:23:30.240139 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.219462 3668 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.252007 3665 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.252344 3701 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.252928 3701 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.212455 3681 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.255810 3681 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.256480 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.257562 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.245867 3662 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.251056 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.264650 1843 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.217810 3664 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.265606 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.266227 2587 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:30.266839 3664 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.267017 3670 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.258960 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.267742 1850 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.267036 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:30.268471 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.269141 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.270020 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.270134 1848 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.271517 3662 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.271564 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.272135 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.272794 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.273340 1921 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:30.274228 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.274806 1924 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:30.274947 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.248786 3671 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.275667 3671 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.277338 3692 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.277756 3692 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.274266 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.253779 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.285239 3696 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.261863 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.258366 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.263358 3677 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.276026 3672 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.263442 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.231068 3680 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.217128 3689 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.285727 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.278143 3692 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.276067 3671 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.292012 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.331413 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.363487 2584 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:30.286087 3690 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.326678 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.287506 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.327231 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.313616 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.376156 3698 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.376492 1843 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:30.364517 1849 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:30.317435 2581 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:30.363062 2582 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:30.365048 3672 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.286453 3697 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:30.363554 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 229798us
I20250626 08:23:30.283037 1847 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.365700 3680 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.327008 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:30.317056 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.317951 3667 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.328567 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.315003 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.278143 3700 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.371383 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.371699 3560 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:30.400559 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 113143us
I20250626 08:23:30.401233 1812 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.376694 3698 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.402730 3662 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.405256 3662 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.406248 3698 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.407140 3662 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.407670 3662 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.407930 3698 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.409404 1814 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.409148 3566 consensus_queue.cc:1046] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:30.410478 3566 raft_consensus.cc:3053] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:30.410843 3566 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:30.410710 3663 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:30.381434 3688 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.381951 3693 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.382565 3704 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:30.383174 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 108783us
I20250626 08:23:30.383117 3668 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.386188 3689 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.383962 3690 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.286814 3699 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.278622 3701 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.393949 1845 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:30.369896 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.437041 1924 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:30.408687 3698 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.396143 3679 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.397022 3680 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.400672 3667 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.372869 1846 raft_consensus.cc:2391] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:30.401813 2585 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:30.403597 3560 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:30.403787 1890 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.403980 3664 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:30.404171 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.441277 3664 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.441751 3664 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:30.443737 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:30.444983 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:30.285810 3696 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.446148 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:30.446411 3677 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.446885 3677 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.447291 3677 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.447798 3696 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.448621 3663 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.449138 3663 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.449649 3663 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.450287 1938 tablet_copy_service.cc:140] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Received BeginTabletCopySession request for tablet ac6c556e015045ac949f4ee3d2048b56 from peer be56082cacb54e9aa8f1a9ee91e70d76 ({username='slave'} at 127.0.0.1:58462)
I20250626 08:23:30.450901 1938 tablet_copy_service.cc:161] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Beginning new tablet copy session on tablet ac6c556e015045ac949f4ee3d2048b56 from peer be56082cacb54e9aa8f1a9ee91e70d76 at {username='slave'} at 127.0.0.1:58462: session id = be56082cacb54e9aa8f1a9ee91e70d76-ac6c556e015045ac949f4ee3d2048b56
I20250626 08:23:30.451599 2550 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.452600 3560 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.452972 3560 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.453331 3560 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.427929 3704 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.455684 3704 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.411859 3566 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.419303 3705 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:30.407980 3662 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.424197 1814 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.422652 3688 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.428047 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.428666 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:30.400936 3697 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.393380 1925 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:30.461350 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:30.468226 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44086.2us
I20250626 08:23:30.438825 3666 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.475046 1850 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:30.474618 1847 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:30.468137 1920 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:30.417573 3700 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.479300 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.480021 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.475723 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:30.479779 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.481312 1841 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:30.433869 3693 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.482519 3680 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.487210 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:30.490561 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.493546 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.436339 3689 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.495755 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.484390 1893 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.467657 3662 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.475710 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:30.500250 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.501253 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.463230 3701 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.501884 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.500623 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.507776 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.508584 2001 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.478783 3667 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:30.478586 3704 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:30.474601 3705 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:30.514797 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.501576 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.496135 3566 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.535028 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.534617 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.539846 1999 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:30.440033 3699 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.500804 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:30.467334 3679 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.543320 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.544530 3708 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.545056 3695 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:30.545993 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:30.511493 3696 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.500701 3707 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.514889 3667 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.496134 3680 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:30.513263 3668 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.515008 3701 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:30.559303 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:30.695176 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 219535us
I20250626 08:23:30.735325 1814 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.536456 3579 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250626 08:23:30.744302 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 278294us
I20250626 08:23:30.538007 3704 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.559818 3666 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.577839 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.585011 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:30.698107 3710 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.738919 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.740093 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.740396 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.579927 1842 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:30.741529 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:30.782267 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 297955us
I20250626 08:23:30.782900 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:30.653872 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 217093us
I20250626 08:23:30.745848 1814 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:30.775274 1850 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:30.741326 3560 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.775832 2551 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.778978 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.777159 1996 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:30.787822 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:30.781095 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 266869us
I20250626 08:23:30.754413 3667 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:30.744225 3579 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.790930 1893 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:30.777966 3666 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.655392 3690 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.758399 3670 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.762907 3668 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.794416 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.794633 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.795007 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.795490 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.796000 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.590870 3709 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.652887 3706 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.750512 3696 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.800827 1893 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:30.810781 3684 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:30.811196 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.811518 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.812527 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.791738 3669 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.757579 3697 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.791931 3691 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.857363 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.792642 3689 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.786579 1994 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:30.800256 3698 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:30.865434 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67991.9us
I20250626 08:23:30.800130 3579 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:30.879997 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.882189 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:30.884840 1938 tablet_copy_source_session.cc:215] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 08:23:30.885641 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.800557 3700 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.776996 3692 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.895098 1846 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
W20250626 08:23:30.899792 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 108378us
I20250626 08:23:30.915331 1814 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:30.858533 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 57819.1us
W20250626 08:23:30.890699 1816 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.552s (SENT)
I20250626 08:23:30.901207 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:30.917315 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.591058 3708 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.857221 3690 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.813695 3688 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.808966 3699 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.924445 3679 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.909402 3698 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250626 08:23:30.924808 1814 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.925045 3711 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:30.925199 3692 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:30.791822 3665 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.806522 3670 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:30.911588 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:30.911736 3702 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:30.911955 3712 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:30.912107 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:30.912602 3713 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.857808 3693 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.877630 3683 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:30.806334 3668 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.879781 3669 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.946862 3681 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250626 08:23:30.920619 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 126309us
I20250626 08:23:30.886608 3691 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:30.924077 2608 debug-util.cc:398] Leaking SignalData structure 0x7b0800333c20 after lost signal to thread 690
W20250626 08:23:30.982745 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080032bda0 after lost signal to thread 1781
W20250626 08:23:30.984400 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080029ee00 after lost signal to thread 1875
W20250626 08:23:30.985898 2608 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb760 after lost signal to thread 1952
I20250626 08:23:30.805826 3710 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.943552 3678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac6c556e015045ac949f4ee3d2048b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 08:23:30.946326 2002 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6"
dest_uuid: "755d83ca0913406b816e95415d502055"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:36848
I20250626 08:23:30.763536 3707 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.947955 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:30.782423 3705 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.951529 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:30.926544 3676 consensus_queue.cc:1046] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20250626 08:23:30.920037 2551 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:30.938797 3695 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:30.978780 3668 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.945596 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.951135 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:30.981259 1918 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 2: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:30.978399 3669 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:30.988200 3691 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:30.987349 2608 debug-util.cc:398] Leaking SignalData structure 0x7b080018ef80 after lost signal to thread 2027
I20250626 08:23:30.991132 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:31.003059 2002 raft_consensus.cc:604] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:31.007253 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 59366.6us
I20250626 08:23:31.007520 2584 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 2: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:30.922901 3684 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:30.955281 3702 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:30.989928 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:30.928877 3690 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:30.946115 3670 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:30.983592 3677 consensus_meta.cc:287] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.536s user 0.010s sys 0.003s
I20250626 08:23:30.947499 3665 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:30.948372 3682 consensus_queue.cc:1046] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20250626 08:23:30.948493 3664 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:31.018890 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 94171.9us
I20250626 08:23:31.021327 3701 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:30.979426 3681 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.031008 3680 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:31.030441 3684 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.027601 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.017979 3662 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:31.042376 2608 debug-util.cc:398] Leaking SignalData structure 0x7b08003111c0 after lost signal to thread 2611
I20250626 08:23:31.064488 3710 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.064811 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.065723 3708 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.070675 1925 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:31.068395 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:30.888402 3709 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.076254 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.076388 1845 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:30.890333 3706 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.129313 1848 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:31.019582 3695 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.084437 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:31.129601 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 92444.6us
I20250626 08:23:31.026803 3704 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.086686 3714 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:31.130852 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.022708 3676 raft_consensus.cc:3053] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:31.073076 3702 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.031709 3560 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:31.022950 3672 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.093621 1814 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:31.013510 3663 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.563s user 0.012s sys 0.000s
I20250626 08:23:31.080765 3665 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:30.929899 3711 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.105798 1849 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:31.090359 3681 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.090473 1993 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:31.114428 3667 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:31.115479 3707 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.115589 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.119786 3666 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.082479 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:30.977337 3712 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.123559 3696 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.105772 3684 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:31.125847 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 205881us
I20250626 08:23:31.086288 3694 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:31.086823 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:31.135414 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 347994us
I20250626 08:23:31.003844 3713 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:31.150420 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 82109.1us
I20250626 08:23:31.182008 2551 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.150545 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.137537 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.185549 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.080978 3677 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.154826 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:31.186192 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48802.3us
I20250626 08:23:31.155426 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.186892 1967 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.187912 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.155997 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.188809 3666 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:31.188992 3667 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:31.189488 1850 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:31.189790 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.190197 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.190397 2001 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:31.190845 1994 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:31.191136 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.191331 3677 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.191895 1921 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:31.158828 3668 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.159682 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.136857 3676 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:31.189006 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:31.161685 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 68179.8us
I20250626 08:23:31.162213 3721 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.167836 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.167147 2000 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:31.137123 3682 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.134696 3695 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:31.134552 3693 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:31.168092 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 277398us
W20250626 08:23:31.163430 3671 consensus_meta.cc:287] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.773s user 0.008s sys 0.002s
I20250626 08:23:31.153216 3714 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:31.141091 3709 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:31.179796 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.180852 1847 raft_consensus.cc:2391] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:31.163375 3679 consensus_queue.cc:1046] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250626 08:23:31.197098 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.197405 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.150346 3708 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250626 08:23:31.164916 3663 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.198155 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:31.234609 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.220633 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.229525 3715 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:31.203784 1893 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:31.198385 3692 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.239012 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.238150 2584 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:31.238673 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.333346 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.246377 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.195097 3662 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.202973 1967 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.236756 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.323187 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.246006 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.336798 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.336148 1842 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:31.200284 3696 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.340880 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.232674 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.191339 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.312374 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.232276 1996 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:31.343385 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.204856 1814 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.309446 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.326018 1846 raft_consensus.cc:1238] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:31.331174 3714 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:31.335343 3719 consensus_queue.cc:1046] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250626 08:23:31.336120 2002 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:23:31.338776 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 135043us
I20250626 08:23:31.312011 1998 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:31.344659 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 139909us
I20250626 08:23:31.311174 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.216250 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:31.347616 1814 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:31.350780 1814 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.351810 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.351938 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:31.352725 1843 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
W20250626 08:23:31.352878 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 149989us
I20250626 08:23:31.216527 3720 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.245038 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.238502 3724 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:31.352937 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 43578.5us
I20250626 08:23:31.217834 3694 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.369881 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.237493 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.354209 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.361855 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.363969 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:31.237735 3697 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20250626 08:23:31.205504 3690 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.338238 3664 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:31.347666 1965 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.353468 3667 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:31.339813 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.369246 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.341502 3672 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.369716 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.238005 3707 consensus_queue.cc:1046] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20250626 08:23:31.353972 3705 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:31.365285 3718 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:31.367111 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.353386 3682 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20250626 08:23:31.380533 1968 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.355715 3710 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.365090 3726 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:31.364604 3709 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.340258 3671 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.355647 3715 raft_consensus.cc:695] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:31.381201 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:31.390178 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.371495 3694 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:31.392431 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 49898us
I20250626 08:23:31.350617 3721 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.446897 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.394380 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:31.447957 1919 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:31.395428 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:31.395609 1850 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:31.396502 1995 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:31.388680 1994 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:31.338040 3676 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.382098 3666 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:31.437700 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:31.388597 3705 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:31.453460 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 72984.4us
I20250626 08:23:31.460862 1848 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:31.398456 3696 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.449787 3718 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.382359 3724 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:31.464469 3689 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:31.543921 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.523878 1924 raft_consensus.cc:1238] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
W20250626 08:23:31.524200 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 66625.5us
I20250626 08:23:31.552424 1746 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:31.454923 3726 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.552506 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.776217 2002 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:31.471820 3666 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:31.537446 3705 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:31.465070 3692 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.324s
I20250626 08:23:31.483549 3706 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.732143 3676 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.775380 3704 raft_consensus.cc:3053] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
W20250626 08:23:31.802721 3728 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.259s user 0.005s sys 0.000s
I20250626 08:23:31.815937 3662 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:31.745831 3715 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.830214 3662 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.796084 3666 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:31.814407 3704 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:31.833730 3662 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.810922 3712 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:31.524372 3679 consensus_queue.cc:1046] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.319s
I20250626 08:23:31.782122 3726 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.793707 3713 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:31.816581 3671 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:31.835088 1846 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:31.835068 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.836954 3704 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.854667 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.840536 3711 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:31.846879 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 303070us
I20250626 08:23:31.852800 3703 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:31.849367 3706 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.844687 3718 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.858525 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.858448 2001 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:31.861424 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.862542 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.863361 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.838167 3668 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:31.867027 1965 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.871750 1965 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.877463 1965 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.881768 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.883419 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:31.885494 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 276831us
I20250626 08:23:31.859926 3683 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:31.551254 3709 consensus_queue.cc:1046] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.322s
I20250626 08:23:31.886965 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.887566 1996 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:31.888332 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.889245 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.890391 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.891144 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.891880 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.892642 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.893055 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:31.897982 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.875823 3712 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:31.904386 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.905831 2585 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:31.901484 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.884831 3713 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.882647 3689 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.886665 3696 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:31.910898 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.891316 3683 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:31.865217 3703 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:31.927278 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.924044 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.907651 3667 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250626 08:23:31.902429 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 205) took 1008 ms. Trace:
I20250626 08:23:31.911317 3710 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:31.949491 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.919760 3704 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:31.918599 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:31.955240 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:31.919275 3708 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20250626 08:23:31.956008 3681 consensus_meta.cc:287] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.787s user 0.012s sys 0.001s
W20250626 08:23:32.001807 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 70128.1us
I20250626 08:23:32.003547 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:31.938863 3709 consensus_queue.cc:1046] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20250626 08:23:32.007200 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:32.011579 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:31.956456 3696 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
W20250626 08:23:32.005841 3579 consensus_meta.cc:287] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.089s user 0.016s sys 0.002s
W20250626 08:23:32.018216 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:32.022306 3734 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20250626 08:23:32.022440 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.022385 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.022460 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.026073 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:32.026537 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.027110 1921 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:32.027855 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.028383 1921 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:32.028945 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:32.025112 1919 consensus_meta.cc:287] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.558s user 0.006s sys 0.001s
I20250626 08:23:32.029687 1919 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:32.030103 1968 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.031000 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:31.946707 2000 rpcz_store.cc:276] 0626 08:23:30.885006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:30.885278 (+ 272us) service_pool.cc:225] Handling call
0626 08:23:31.884670 (+999392us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400078ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:31.893270 (+ 8600us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":593371520}
I20250626 08:23:32.031626 1928 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:32.032115 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:32.013062 3714 consensus_meta.cc:287] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.657s user 0.013s sys 0.027s
I20250626 08:23:32.033241 3714 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.023855 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.035168 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.035356 3714 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.036646 1918 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250626 08:23:32.038937 3663 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.032416 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.030591 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.053627 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:31.996382 3681 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.061272 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.056325 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.067428 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:32.068243 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:32.069305 3670 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.963s user 0.010s sys 0.000s
I20250626 08:23:32.069895 3670 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.073601 3697 raft_consensus.cc:3053] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:31.996523 3683 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.075093 1923 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
W20250626 08:23:32.075227 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 63728.6us
I20250626 08:23:32.064924 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.078312 2585 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.074139 3697 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.077046 1926 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:31.995610 3711 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20250626 08:23:32.086242 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 56281.3us
I20250626 08:23:32.085595 3706 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.087883 3697 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:32.088838 3695 consensus_meta.cc:287] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.766s user 0.004s sys 0.016s
I20250626 08:23:32.089224 3695 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.088276 3683 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.083318 1923 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:32.090664 1841 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:32.091341 1844 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.092605 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:32.093991 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 86749.6us
W20250626 08:23:32.063938 3669 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.988s user 0.012s sys 0.000s
I20250626 08:23:32.097524 1968 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.039685 3579 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:32.102069 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 576579us
I20250626 08:23:32.115051 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:32.115738 1848 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:32.116791 1848 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.118337 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:32.122365 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 113749us
I20250626 08:23:32.124344 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.092437 3696 consensus_queue.cc:1230] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer 548099d81e7f46ac9b2f03d9e5e678a6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250626 08:23:32.085790 3666 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.114356 3670 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:32.107903 3724 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:32.121486 3713 consensus_queue.cc:1046] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20250626 08:23:32.106848 3669 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:32.232282 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 134832us
W20250626 08:23:32.232580 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.238649 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1796) took 1110 ms. Trace:
I20250626 08:23:32.238943 1998 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:32.239619 1927 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:32.241735 3695 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.087896 3706 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.266495 3669 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.267416 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.268333 3579 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.268828 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1779) took 1319 ms. Trace:
I20250626 08:23:32.269202 1997 rpcz_store.cc:276] 0626 08:23:30.949863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:30.973874 (+ 24011us) service_pool.cc:225] Handling call
0626 08:23:31.886877 (+913003us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:31.887034 (+ 157us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:31.887059 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:31.887098 (+ 39us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:31.887126 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:31.887173 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:32.268643 (+381470us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400078ad0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:32.268742 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2067479296}
I20250626 08:23:32.271054 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.271441 3678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36150 (request call id 487) took 2401 ms. Trace:
I20250626 08:23:32.271610 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.272460 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.272724 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:32.272959 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.273172 1919 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:32.273041 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:32.242403 3665 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.069s user 0.005s sys 0.018s
W20250626 08:23:32.257352 3702 consensus_meta.cc:287] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.104s user 0.006s sys 0.013s
I20250626 08:23:32.243364 3705 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:32.244334 3737 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.122s user 0.001s sys 0.008s
I20250626 08:23:32.285199 1968 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:32.244515 3718 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.245229 3681 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.241829 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1525) took 1211 ms. Trace:
I20250626 08:23:32.107733 2581 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:32.247761 3738 raft_consensus.cc:3053] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:23:32.247962 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 115858us
I20250626 08:23:32.249208 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1526) took 1182 ms. Trace:
I20250626 08:23:32.254025 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1621) took 1473 ms. Trace:
W20250626 08:23:32.251847 3694 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.727s user 0.007s sys 0.014s
W20250626 08:23:32.251525 3684 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.019s user 0.008s sys 0.010s
I20250626 08:23:32.259373 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.263619 2001 rpcz_store.cc:276] 0626 08:23:31.128314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.191124 (+ 62810us) service_pool.cc:225] Handling call
0626 08:23:32.234439 (+1043315us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54000799d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:32.238584 (+ 4145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1185214848}
I20250626 08:23:32.278860 1843 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.271553 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.255865 3689 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.281342 3693 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.380096 3693 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.282344 3702 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.293843 3706 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:32.380834 3706 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.380923 3693 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.262238 3703 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.290458 3709 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:32.382182 3693 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.382230 3709 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.383023 3706 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.290644 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.291074 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:32.292222 3666 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.292804 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.293519 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:32.351884 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 66788.6us
I20250626 08:23:32.375844 1848 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:32.371620 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.294275 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.334834 1995 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:32.377832 1846 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250626 08:23:32.392431 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.393568 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.376052 1924 rpcz_store.cc:276] 0626 08:23:30.780518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:30.780791 (+ 273us) service_pool.cc:225] Handling call
0626 08:23:30.789846 (+ 9055us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:31.835022 (+1045176us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.253759 (+418737us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b5400260f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.253961 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1595984768}
I20250626 08:23:32.286777 3665 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.294216 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.396565 1849 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.391829 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.399005 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.393760 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.400350 1848 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:32.343884 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48829.6us
I20250626 08:23:32.376717 3724 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:32.384024 1845 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:32.383497 1923 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.331418 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.315559 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:32.378733 1841 rpcz_store.cc:276] 0626 08:23:31.030320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.368913 (+338593us) service_pool.cc:225] Handling call
0626 08:23:31.369191 (+ 278us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.092828 (+723637us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.241619 (+148791us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54002f21d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.241775 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1658819456}
I20250626 08:23:32.378386 1844 rpcz_store.cc:276] 0626 08:23:31.066360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.370950 (+304590us) service_pool.cc:225] Handling call
0626 08:23:31.388944 (+ 17994us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.093044 (+704100us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.248983 (+155939us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b54002ba2d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.249150 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1615010560}
I20250626 08:23:32.388289 1967 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:32.393568 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:32.401768 1814 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:32.377174 3691 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.298s user 0.011s sys 0.030s
I20250626 08:23:32.331718 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.405795 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:32.408113 1850 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.262367 3726 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.411512 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1543) took 1060 ms. Trace:
I20250626 08:23:32.412508 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.410866 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:32.376024 3738 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.419618 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 302) took 1109 ms. Trace:
I20250626 08:23:32.419981 1843 rpcz_store.cc:276] 0626 08:23:31.310147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.775710 (+465563us) service_pool.cc:225] Handling call
0626 08:23:31.800423 (+ 24713us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.378158 (+577735us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.419383 (+ 41225us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54000343d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.419546 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1091212544}
I20250626 08:23:32.414319 1967 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:32.422955 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.413807 1850 rpcz_store.cc:276] 0626 08:23:31.350683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:32.386944 (+1036261us) service_pool.cc:225] Handling call
0626 08:23:32.407957 (+ 21013us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.411274 (+ 3317us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.411452 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:32.425721 1847 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.426811 1844 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:32.415392 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:32.427335 1967 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:32.421105 1848 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:32.428687 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:32.430068 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.431037 2586 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.432473 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.434841 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.408741 3724 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:32.403162 3579 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.436281 1893 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.421231 3671 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.384207 3694 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.444151 1893 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.385890 3684 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.418545 3714 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:32.422501 3722 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.415474 3702 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.409282 3703 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.418855 3681 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.428145 3706 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.438022 3733 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.442768 3724 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.421994 3708 consensus_queue.cc:1046] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250626 08:23:32.426018 3711 consensus_queue.cc:1046] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250626 08:23:32.430024 3689 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.383159 3709 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:32.436388 3719 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:32.453996 3670 consensus_queue.cc:1046] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:32.454757 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1625) took 1661 ms. Trace:
I20250626 08:23:32.437954 3666 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:32.463197 1848 raft_consensus.cc:1238] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.426402 3690 raft_consensus.cc:3053] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.438284 2578 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:32.443957 3676 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.442623 3735 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.493505 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1811) took 1010 ms. Trace:
I20250626 08:23:32.493892 1995 rpcz_store.cc:276] 0626 08:23:31.483287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.867416 (+384129us) service_pool.cc:225] Handling call
0626 08:23:32.493347 (+625931us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400298954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:32.493446 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1018777984}
I20250626 08:23:32.494251 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.494974 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.495503 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.496060 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.496667 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.497115 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.498210 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.498309 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.499454 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.499722 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 303) took 1158 ms. Trace:
I20250626 08:23:32.500078 1847 rpcz_store.cc:276] 0626 08:23:31.341333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.775021 (+433688us) service_pool.cc:225] Handling call
0626 08:23:31.795707 (+ 20686us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.426265 (+630558us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.499492 (+ 73227us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54002562d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.499667 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1430167936}
I20250626 08:23:32.271869 3678 rpcz_store.cc:276] 0626 08:23:29.869676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:29.881012 (+ 11336us) service_pool.cc:225] Handling call
0626 08:23:29.966081 (+ 85069us) ts_tablet_manager.cc:928] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
0626 08:23:30.941753 (+975672us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:32.033104 (+1091351us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:32.033389 (+ 285us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:32.270649 (+237260us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:32.270949 (+ 300us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:32.271298 (+ 349us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:32.271358 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":62367,"tablet-copy.queue_time_us":83813,"thread_start_us":142247,"threads_started":2}
I20250626 08:23:32.501644 3678 tablet_copy_client.cc:806] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 0 data blocks...
I20250626 08:23:32.502081 3709 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.502277 3678 tablet_copy_client.cc:670] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Starting download of 1 WAL segments...
I20250626 08:23:32.446581 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:32.420217 3691 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.413630 3749 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20250626 08:23:32.477209 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48615.7us
W20250626 08:23:32.477510 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.449097 3714 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:32.466763 3579 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.458865 3697 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:32.485198 1927 rpcz_store.cc:276] 0626 08:23:30.793785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:30.793943 (+ 158us) service_pool.cc:225] Handling call
0626 08:23:30.794207 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.263813 (+1469606us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.454468 (+190655us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003679d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.454687 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3111547648}
I20250626 08:23:32.461278 3715 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20250626 08:23:32.487016 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.472515 3704 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:32.491607 3670 raft_consensus.cc:3053] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.473407 3727 raft_consensus.cc:3053] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:32.506353 1967 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:32.510332 3744 consensus_queue.cc:1046] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:32.510787 3749 raft_consensus.cc:3053] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.511893 3691 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.513218 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.516464 3708 consensus_queue.cc:1046] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:32.518209 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.486414 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.522360 1968 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.525997 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.487807 3719 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:32.529870 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.529018 3726 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.531525 3684 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.489717 3690 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.508827 3666 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:32.517242 3712 consensus_queue.cc:1046] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:32.512594 3714 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.535837 1843 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.546790 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.536051 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.519125 3705 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.535200 3711 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:32.527009 3704 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:32.543572 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57247.1us
I20250626 08:23:32.542284 3719 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.475513 3738 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.526815 3670 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:32.529688 3727 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:32.533481 3749 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.539918 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.562475 1998 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:32.545136 3695 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.544513 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.565562 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.558360 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.555167 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.570504 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.570803 3739 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.542395 3718 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.547912 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.549343 3708 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:32.570684 3679 consensus_queue.cc:1046] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:32.569839 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.556036 3678 tablet_copy_client.cc:538] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 08:23:32.562537 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.551041 3711 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:32.576264 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.552999 3665 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.556486 3706 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.576334 1927 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:32.556593 3704 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.579123 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.657326 3753 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:32.643607 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 73853.6us
I20250626 08:23:32.571681 3707 raft_consensus.cc:3053] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.578280 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:32.630734 3708 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.654805 3681 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.658210 3711 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.667695 3707 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.672441 1999 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:32.674823 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:32.670033 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 118322us
I20250626 08:23:32.658421 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:32.658063 3694 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:32.719183 1967 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:32.733716 1967 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:32.739183 1812 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:32.741288 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.735064 1918 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:32.654594 3690 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.754539 3761 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.757045 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.730283 3696 raft_consensus.cc:3053] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:23:32.767638 3758 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250626 08:23:32.735774 3712 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.767021 3696 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.783881 3696 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.789770 3696 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.790736 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.673816 3666 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.674369 3727 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.810452 3707 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.811858 3684 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:32.815150 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.816733 3689 raft_consensus.cc:3053] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:32.763698 3708 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.680210 3749 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.761646 3706 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.074s
W20250626 08:23:32.833568 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46491.9us
I20250626 08:23:32.833722 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.674012 3670 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.839644 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1524) took 1804 ms. Trace:
W20250626 08:23:32.847344 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 403242us
W20250626 08:23:32.888530 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 52299.6us
I20250626 08:23:32.888695 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.888291 1849 rpcz_store.cc:276] 0626 08:23:31.029482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.353589 (+324107us) service_pool.cc:225] Handling call
0626 08:23:31.371121 (+ 17532us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.397022 (+1025901us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.827864 (+430842us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002bacd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:32.833658 (+ 5794us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2357919488}
I20250626 08:23:32.840390 3761 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.748858 3695 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.890545 1893 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.890870 3749 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.846385 3689 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:32.890897 3710 raft_consensus.cc:3053] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:32.849728 2000 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:32.891619 3710 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:23:32.892323 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:32.893011 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 43705.4us
I20250626 08:23:32.895036 1893 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.835031 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.840871 3726 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:32.847520 3727 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:32.899895 3706 consensus_queue.cc:1230] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Peer 548099d81e7f46ac9b2f03d9e5e678a6 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250626 08:23:32.928876 2582 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:32.906450 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:32.933001 1849 raft_consensus.cc:604] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:32.908802 1843 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:32.935771 3724 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.936393 1923 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:32.937018 1923 raft_consensus.cc:604] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:32.841019 1993 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:32.913705 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.938401 2001 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:32.938757 3453 raft_consensus.cc:3053] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:32.939152 3453 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:32.905745 1893 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.914463 3710 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:32.953859 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 50828.7us
I20250626 08:23:32.892031 3689 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.894882 3676 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.848395 3684 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.898413 3673 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:32.921880 3755 log.cc:927] Time spent T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20250626 08:23:32.894409 3735 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:32.936218 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 516) took 1048 ms. Trace:
I20250626 08:23:32.913137 3704 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:32.942407 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.945437 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:32.947080 3566 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:32.949885 1816 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.925613 3678 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap starting.
I20250626 08:23:32.898186 3707 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:32.955420 3712 consensus_queue.cc:1046] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:32.959460 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:32.980022 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 92317us
I20250626 08:23:32.980370 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:32.962129 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.966012 2582 rpcz_store.cc:276] 0626 08:23:31.888157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.888292 (+ 135us) service_pool.cc:225] Handling call
0626 08:23:32.237358 (+349066us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:32.928662 (+691304us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:32.929365 (+ 703us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:32.936006 (+ 6641us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400045950. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:32.936133 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1596617728}
W20250626 08:23:32.977185 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 220195us
I20250626 08:23:32.970798 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.971400 3684 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:32.982502 1923 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:32.971838 3710 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:32.972301 3763 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.972518 3453 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:32.970883 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:32.974095 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.974014 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.957896 3747 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.981097 1816 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:32.982156 1967 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.982578 3724 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:32.982585 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:32.986210 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:32.988431 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:32.989666 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:32.900139 3726 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:32.992415 3693 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.610s user 0.010s sys 0.002s
I20250626 08:23:33.004348 3738 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:32.994313 3579 raft_consensus.cc:991] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: : Instructing follower 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:33.001569 3697 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:32.986812 3718 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.002061 3721 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:32.985142 3712 raft_consensus.cc:3053] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:33.005153 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.009382 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:33.014101 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1506) took 1271 ms (client timeout 1530 ms). Trace:
I20250626 08:23:33.015642 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:32.998572 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.018298 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.018951 1967 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.020632 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.021389 1848 raft_consensus.cc:2391] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:33.014631 1993 rpcz_store.cc:269] 0626 08:23:31.743111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:32.267120 (+524009us) service_pool.cc:225] Handling call
0626 08:23:33.009150 (+742030us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b5400203fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:33.014040 (+ 4890us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1266424704}
I20250626 08:23:33.021561 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.021399 2002 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:33.026329 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.026893 1848 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:33.004395 3726 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.028579 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.034965 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.035786 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.036500 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.037133 1849 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:33.035986 1842 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:33.009840 3712 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:33.016018 3673 raft_consensus.cc:1079] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Signalling peer 548099d81e7f46ac9b2f03d9e5e678a6 to start an election
I20250626 08:23:33.045054 2578 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:33.046535 2586 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:33.047014 3690 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.038288 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:33.047489 3709 consensus_meta.cc:287] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.545s user 0.007s sys 0.003s
I20250626 08:23:33.047946 1847 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:33.049120 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.041216 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.042095 1843 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:33.043154 3754 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.043334 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.044274 3697 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:33.044150 3712 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.044827 3704 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.000082 3669 consensus_queue.cc:1046] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20250626 08:23:33.040333 1816 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.024559 3759 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.064538 3731 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.065536 3666 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.052425 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.066926 3749 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:33.068266 3689 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:33.061239 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 72544.6us
I20250626 08:23:33.070991 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.072363 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.054767 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.006139 3684 consensus_queue.cc:1046] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250626 08:23:33.074741 1997 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:33.075067 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.075731 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.076922 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.078078 3693 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:33.078548 3684 raft_consensus.cc:3053] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:33.078971 3696 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.079030 3684 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.079413 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.079739 3696 raft_consensus.cc:3053] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:33.080165 3696 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:33.079931 3684 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.081346 3684 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.027374 3724 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:33.081838 3724 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.082290 3724 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:33.082479 3670 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.083107 1924 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:33.056192 3697 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.084622 3697 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:33.061069 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 52782.8us
I20250626 08:23:33.088135 1746 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32850 (request call id 305) took 1689 ms. Trace:
I20250626 08:23:33.088793 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.063279 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.091087 1816 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.091301 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.092548 1816 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.080957 3696 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.093991 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:33.094379 1999 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:33.056560 1923 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:33.110065 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:33.051604 3718 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:33.053385 3706 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.110446 1816 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.110065 3716 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.112061 3696 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:33.088142 3714 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.000s
I20250626 08:23:33.116180 2000 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:33.116521 3453 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.117172 3712 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:33.118371 3719 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.547s user 0.011s sys 0.000s
I20250626 08:23:33.119396 3754 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.148478 1968 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.127998 3744 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.078663 1918 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.119133 3681 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.110304 3666 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.095220 3711 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.127961 1996 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:33.113083 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.128475 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:33.128926 1816 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.131647 1967 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.131202 3721 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:33.160219 2586 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:33.088562 1746 rpcz_store.cc:276] 0626 08:23:31.398572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:31.398897 (+ 325us) service_pool.cc:225] Handling call
0626 08:23:33.088041 (+1689144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:31.888118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:31.888694 (+ 576us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:31.888727 (+ 33us) write_op.cc:435] Acquired schema lock
0626 08:23:31.888741 (+ 14us) tablet.cc:660] Decoding operations
0626 08:23:31.889080 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:31.889163 (+ 83us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:31.889191 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0626 08:23:31.889406 (+ 215us) tablet.cc:699] Row locks acquired
0626 08:23:31.889419 (+ 13us) write_op.cc:260] PREPARE: finished
0626 08:23:31.889568 (+ 149us) write_op.cc:270] Start()
0626 08:23:31.889682 (+ 114us) write_op.cc:276] Timestamp: P: 1750926211889522 usec, L: 0
0626 08:23:31.889698 (+ 16us) op_driver.cc:348] REPLICATION: starting
0626 08:23:31.890389 (+ 691us) log.cc:844] Serialized 536 byte log entry
0626 08:23:32.445556 (+555167us) op_driver.cc:464] REPLICATION: finished
0626 08:23:32.663819 (+218263us) write_op.cc:301] APPLY: starting
0626 08:23:32.668062 (+ 4243us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:32.674458 (+ 6396us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:32.740794 (+ 66336us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:32.783010 (+ 42216us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:32.783171 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 08:23:32.783217 (+ 46us) write_op.cc:312] APPLY: finished
0626 08:23:32.783443 (+ 226us) log.cc:844] Serialized 23 byte log entry
0626 08:23:32.783619 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:32.783700 (+ 81us) write_op.cc:454] Released schema lock
0626 08:23:32.783741 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0626 08:23:32.784130 (+ 389us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:32.784162 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":149747,"apply.run_cpu_time_us":2233,"apply.run_wall_time_us":125975,"num_ops":1,"prepare.queue_time_us":56026,"prepare.run_cpu_time_us":38152,"prepare.run_wall_time_us":102893,"raft.queue_time_us":156187,"raft.run_cpu_time_us":20335,"raft.run_wall_time_us":99700,"replication_time_us":555683,"thread_start_us":337549,"threads_started":4,"wal-append.queue_time_us":190848}]]}
I20250626 08:23:33.164700 1816 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.155220 3706 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:33.125543 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.120476 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:23:33.178337 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 57933.2us
I20250626 08:23:33.179095 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.179941 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.180701 2551 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.182016 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:33.183036 3768 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20250626 08:23:33.183660 3744 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.185099 1890 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:33.143600 3714 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.138106 3693 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:33.187321 3693 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:33.187884 3697 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.188442 3708 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:33.190753 2001 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:33.192008 2551 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.142887 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.192816 1999 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:33.193058 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:23:33.168843 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 143040us
I20250626 08:23:33.131108 3724 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.171325 1928 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250626 08:23:33.145550 3716 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20250626 08:23:33.204653 1997 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:33.205401 3724 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.172048 3767 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.161983 3721 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:33.175863 3711 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.186566 3727 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:33.193499 3669 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:33.209692 3669 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.210410 3669 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.210479 3727 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:33.211011 3727 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.150995 3719 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.177501 3681 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:33.225605 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.229228 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.231547 3714 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:33.193826 3744 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:33.236804 3744 raft_consensus.cc:695] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.237740 3744 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.195868 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.246438 3763 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:33.196164 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.207456 1927 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.178972 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.170372 3754 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.202219 3764 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.255208 3764 raft_consensus.cc:695] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.208355 3721 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:33.209234 3740 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.209821 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.188341 3693 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.202538 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.168778 1816 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.194201 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.264860 1993 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:33.265938 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.266607 1993 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:33.193815 3766 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.213840 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.216094 1994 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:33.216264 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.220060 3697 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.220695 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.278514 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.223477 3719 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.279201 2001 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 2: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:33.279985 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.280539 2001 raft_consensus.cc:2374] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2. Re-sending same reply.
I20250626 08:23:33.280812 1843 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:60120
I20250626 08:23:33.281316 1843 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 08:23:33.257480 3764 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.281674 1843 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.250126 1919 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:33.281330 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.287665 2001 raft_consensus.cc:2391] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:33.256170 3714 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.285620 3754 consensus_queue.cc:1046] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:33.292025 3754 raft_consensus.cc:3053] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:33.292411 3754 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.267551 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:33.260346 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 91658us
I20250626 08:23:33.261742 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.262522 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.263123 3763 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:33.264039 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.302251 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.302145 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.303670 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.275401 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.276957 1997 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:33.277580 3727 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.278028 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.252849 2002 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:33.309521 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:33.309942 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:33.249493 1999 raft_consensus.cc:2374] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3. Re-sending same reply.
I20250626 08:23:33.297071 1816 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.311030 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.280301 3667 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.317976 3738 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:33.313150 3716 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:33.319193 3716 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.319618 3716 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.320987 1998 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:33.321254 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.306440 2551 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.313076 1968 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.307870 3673 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:33.314345 3735 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:33.314381 3579 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:33.315833 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.317610 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.328199 2551 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.331194 1816 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.332834 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.334970 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.335127 1893 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.294081 3719 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.337388 1816 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.329308 1843 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:33.308292 3697 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.345765 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.344553 1893 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.293283 3754 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.349442 1816 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.350816 1843 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:33.310861 3714 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:33.330359 3738 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.331316 3718 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.336673 3763 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.367552 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.351436 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:33.369783 1997 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:33.336572 3735 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:33.339831 3665 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:33.366611 1996 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:33.367543 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:33.373978 1926 raft_consensus.cc:3053] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:33.367776 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.374903 1998 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.341143 3759 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.338418 3766 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.343183 3696 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.356467 3681 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:33.340227 3740 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:33.374329 1926 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.376937 2001 raft_consensus.cc:1238] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250626 08:23:33.347051 3731 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.346107 3693 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.354915 3683 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.380781 1996 raft_consensus.cc:1238] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:33.388192 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.355049 3714 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.392791 1995 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:23:33.369837 3678 tablet_bootstrap.cc:700] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent opening tablet: real 0.123s user 0.001s sys 0.001s
I20250626 08:23:33.364395 3668 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:33.373494 3665 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:33.374771 3735 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.402372 3743 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.347209 3719 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.421775 3743 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.399379 2002 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.379235 3754 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.405316 3667 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:33.414305 3566 consensus_queue.cc:1046] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:33.414742 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.429934 3566 raft_consensus.cc:3053] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:33.430178 2001 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.430358 3566 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.431267 3566 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.432574 3566 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:33.416456 1996 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.419441 3683 raft_consensus.cc:695] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.399564 3714 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.401593 3673 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.391955 1926 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:33.400743 3681 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.418367 3729 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20250626 08:23:33.428755 1995 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:33.428543 3759 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:33.426309 3668 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.394788 3766 raft_consensus.cc:695] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.429934 2002 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:33.417237 2000 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:33.401732 3740 raft_consensus.cc:695] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.450626 1926 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.422763 3665 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.450575 2001 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:33.452226 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.454571 1998 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:33.400699 3669 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.468736 1997 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250626 08:23:33.480026 2586 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.439378 3667 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.489078 3668 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:33.451403 3683 consensus_queue.cc:237] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.479122 3727 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.489306 3740 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.483812 3759 raft_consensus.cc:3053] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:33.487635 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:33.487905 3718 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:33.500834 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:33.495653 3759 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.499091 2578 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.500733 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.500479 3719 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.500464 3708 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.516038 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:33.487223 3766 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.529953 3667 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.490185 3743 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.492776 3681 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.547484 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:33.539528 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 87376.8us
I20250626 08:23:33.528779 3670 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.534567 3719 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:33.549376 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:33.526651 3721 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.570856 3721 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.574388 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.575062 1997 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.574061 3670 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.576822 1842 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:33.533071 3759 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.582005 3759 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.588797 1926 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:33.581689 3716 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.594043 3716 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:33.594107 2582 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:33.552978 3669 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:33.599391 2586 raft_consensus.cc:1238] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:33.601562 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:33.614073 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 247790us
I20250626 08:23:33.622447 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.623462 1848 raft_consensus.cc:2391] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:33.626262 1997 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:33.627739 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:33.531929 3708 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.550642 3696 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.635504 3714 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.635872 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.637293 3714 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:33.593708 3670 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.633313 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.640291 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.641484 1848 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:33.638265 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 87820.4us
I20250626 08:23:33.629609 3702 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:33.644515 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.649576 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:33.653349 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:33.643249 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.637831 3780 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20250626 08:23:33.653939 1816 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:33.716504 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.681175 1742 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:23:33.774636 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 120840us
I20250626 08:23:33.774905 1850 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
W20250626 08:23:33.722138 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 75214.6us
I20250626 08:23:33.776247 1968 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:33.641384 3716 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:33.785544 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.801189 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.802177 1998 raft_consensus.cc:1238] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.723191 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:33.652030 3770 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
W20250626 08:23:33.826078 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 381422us
I20250626 08:23:33.640974 3696 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.802177 1997 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.828581 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.824182 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:33.747570 3738 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:33.724108 3708 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.671725 3666 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:33.654048 3714 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:33.830778 1848 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:33.800189 3784 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:33.654314 3670 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:33.747440 3731 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:33.799746 3781 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.799357 3697 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250626 08:23:33.878643 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 53537.6us
I20250626 08:23:33.650960 3702 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.694995 1994 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:33.800009 3748 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:33.666500 3774 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.170s
I20250626 08:23:33.651059 3668 consensus_queue.cc:1046] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.181s
I20250626 08:23:33.902951 1997 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:33.656366 3667 consensus_queue.cc:1046] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.215s
I20250626 08:23:33.683744 1993 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:33.729231 3763 consensus_queue.cc:1046] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.146s
I20250626 08:23:33.829248 3703 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:33.830430 3716 raft_consensus.cc:695] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:33.929667 3763 raft_consensus.cc:3053] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:33.877789 3714 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:33.934931 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.935462 1848 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:33.892684 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 55052.2us
I20250626 08:23:33.860342 3747 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:33.894434 3697 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.878139 3666 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:33.902371 1845 raft_consensus.cc:1238] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:33.686057 3579 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:33.811985 1918 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:33.946878 3697 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:33.949731 3666 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:33.934002 3716 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.934269 3703 raft_consensus.cc:695] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.954670 3791 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:33.954990 3703 consensus_queue.cc:237] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.955862 3791 raft_consensus.cc:3053] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:33.956247 3791 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:33.936499 3714 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:33.957118 3791 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.903133 3748 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.927759 3696 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.928653 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:33.933524 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.879276 3738 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:33.962347 2550 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.966814 3765 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:33.967250 3765 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:33.967703 3765 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:33.966806 3738 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.973608 2579 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:33.974627 3748 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:33.894961 3665 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:33.975733 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:33.976411 1927 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:33.978813 2550 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:33.982578 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:33.942340 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:33.942516 1926 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:33.902966 3702 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:33.948778 3747 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.930203 3763 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.958433 3667 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:33.958915 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:33.937273 3681 consensus_queue.cc:1046] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20250626 08:23:33.964066 3783 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.238s user 0.000s sys 0.001s
I20250626 08:23:33.985808 1893 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:34.014067 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:34.014660 3775 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:34.015190 3775 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:33.987257 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:33.995512 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.890758 3784 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.004179 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.019794 1847 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:34.004856 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:33.960119 3791 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:33.922073 3670 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:34.010591 1843 raft_consensus.cc:1238] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:34.012101 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 52491.1us
W20250626 08:23:33.998495 3693 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.602s user 0.003s sys 0.008s
I20250626 08:23:34.022293 1926 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:33.998361 3715 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:33.998960 3738 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:34.026736 1928 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:34.031062 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:33.999598 3677 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:34.042534 3665 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:34.042783 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 47424.3us
W20250626 08:23:34.045301 2578 consensus_meta.cc:287] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.517s user 0.010s sys 0.000s
I20250626 08:23:34.045689 2578 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:34.046419 3677 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:34.046087 3763 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.046903 3681 raft_consensus.cc:3053] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:34.047313 3681 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:34.047348 3677 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.048059 3702 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:34.048609 3702 raft_consensus.cc:695] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:34.048198 3681 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.039456 3766 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250626 08:23:34.037963 3715 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:34.051203 3715 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:34.049482 3681 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.038290 3738 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:34.054804 3738 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.047389 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:34.057227 3735 consensus_meta.cc:287] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.606s user 0.006s sys 0.003s
I20250626 08:23:34.057116 1918 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:34.057967 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:34.059800 3697 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.061160 1996 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:34.005446 3667 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:34.062678 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:34.060081 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:34.046670 3764 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:34.064659 1928 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250626 08:23:34.076999 3754 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.640s user 0.005s sys 0.007s
I20250626 08:23:34.078190 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.087806 1843 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
W20250626 08:23:34.086436 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 67797.9us
I20250626 08:23:34.047325 3763 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.016147 3775 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.042623 3784 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.098987 3697 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.087188 3683 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:34.107239 1997 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:34.083537 3781 raft_consensus.cc:695] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:34.118278 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 53107.5us
I20250626 08:23:34.120548 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:34.121330 1890 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:34.122800 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:34.123589 1812 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:34.123536 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.144273 1844 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:34.127411 1843 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:34.146427 1968 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:34.127091 3667 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:34.062271 3566 consensus_queue.cc:1162] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20250626 08:23:34.092370 3693 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:34.150219 1847 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:34.151487 1926 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:34.094352 3747 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.155650 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.156431 2578 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:34.158128 3792 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:34.153450 3685 consensus_queue.cc:1046] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20250626 08:23:34.147228 3754 consensus_queue.cc:1046] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20250626 08:23:34.159945 3685 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:34.160315 3685 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:34.147310 3579 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:34.160874 3579 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:34.161293 3579 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.162184 3681 raft_consensus.cc:3053] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:34.162616 3681 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:34.163425 3681 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.147267 3697 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:34.165282 3716 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.166484 1998 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:34.145288 3665 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:34.167788 3666 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.049506 3702 consensus_queue.cc:237] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.166687 3716 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.170137 2586 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:34.184001 2000 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:34.184458 2001 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:34.161171 3685 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.079285 3735 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.168382 3756 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20250626 08:23:34.197827 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44679.7us
I20250626 08:23:34.154469 3714 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.153975 3731 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250626 08:23:34.149556 3727 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:34.174396 1998 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:34.175202 3695 raft_consensus.cc:3053] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
W20250626 08:23:34.228209 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:34.259479 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 65470.5us
W20250626 08:23:34.271039 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.290251 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:34.290511 3666 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.290952 1923 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:34.279843 3800 consensus_queue.cc:1046] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20250626 08:23:34.196041 3697 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:34.295488 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.154351 3784 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20250626 08:23:34.279080 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:34.298638 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.314671 1923 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:34.316854 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:34.317785 2002 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:34.317675 1814 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:34.294309 3697 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.294482 3714 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:34.281082 3727 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:34.325450 3685 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:34.331323 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 50537.1us
I20250626 08:23:34.333704 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.334188 3735 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.318884 3682 consensus_queue.cc:1046] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250626 08:23:34.335611 3682 raft_consensus.cc:3053] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:34.335958 3682 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:34.336135 3666 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:34.336637 3666 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:34.336880 3682 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.280215 3695 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:34.294941 3777 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:34.226514 3764 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20250626 08:23:34.231176 3748 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20250626 08:23:34.281340 3691 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20250626 08:23:34.400553 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.340857 3727 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.341759 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:34.326208 3683 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20250626 08:23:34.337455 3666 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.295774 3799 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.165s
I20250626 08:23:34.338526 3781 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:34.385253 3777 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.386359 1993 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:34.341698 3774 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20250626 08:23:34.448894 1847 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:34.449759 1842 raft_consensus.cc:1238] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:34.344095 3579 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.471119 1812 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:34.487649 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.488603 1841 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:34.501708 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:34.508989 3780 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.510069 3579 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.480662 3718 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250626 08:23:34.520293 3801 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20250626 08:23:34.515277 3715 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:34.524019 3806 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.091s user 0.003s sys 0.000s
I20250626 08:23:34.524605 1847 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:34.524820 3715 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.516055 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:34.528839 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.528548 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.531546 1847 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:34.532833 1996 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:34.534593 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:34.540045 1968 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:34.526368 3669 consensus_queue.cc:1046] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20250626 08:23:34.544692 3670 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:34.529830 3697 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.545511 3670 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.526373 3716 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.484114 3668 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20250626 08:23:34.572536 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.573429 3810 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250626 08:23:34.579175 3738 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.524s user 0.011s sys 0.000s
I20250626 08:23:34.555155 3453 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:34.541749 3708 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.569985 3715 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20250626 08:23:34.589097 1847 raft_consensus.cc:2391] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:34.564842 3696 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:34.509649 3780 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.580837 3727 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.550784 3670 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.597342 3733 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:34.594988 3708 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.593840 3453 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:34.609660 3453 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:34.595646 3696 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:34.598153 3668 consensus_queue.cc:1230] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250626 08:23:34.613901 3718 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.615309 1841 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:34.608978 3765 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.639s user 0.008s sys 0.005s
I20250626 08:23:34.627790 1890 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:34.595727 3738 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.615483 3696 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:34.567330 3802 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20250626 08:23:34.663178 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.654706 1841 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:34.605674 3733 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:34.691439 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 243854us
I20250626 08:23:34.658077 3777 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.634384 3718 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:34.697254 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 46925us
I20250626 08:23:34.657747 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:34.715114 3733 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.713291 2578 raft_consensus.cc:2366] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:34.718355 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.719528 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:34.720564 1847 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:34.691118 3756 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:34.640673 3765 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:34.730703 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 103019us
I20250626 08:23:34.738363 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.743701 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:34.744488 1923 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:34.745790 2586 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:34.725023 3708 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:34.736660 3763 consensus_meta.cc:287] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.620s user 0.007s sys 0.006s
I20250626 08:23:34.746279 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:34.752336 1927 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:34.728439 3714 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250626 08:23:34.731361 3678 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20250626 08:23:34.754297 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.755136 2582 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:23:34.760783 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.742388 3780 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.793479 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:34.794519 1923 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:34.801496 3798 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.802115 3798 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:34.785784 3827 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20250626 08:23:34.804368 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.788793 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.787734 3800 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.793650 3665 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.791517 3741 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.791206 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.729157 3756 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.826869 1890 leader_election.cc:400] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:34.809805 3727 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.837886 1890 leader_election.cc:403] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250626 08:23:34.842296 2586 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:34.844601 3756 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:34.847761 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.868093 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:34.875746 3791 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.847s user 0.004s sys 0.008s
I20250626 08:23:34.837822 3665 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.900451 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1565) took 1276 ms. Trace:
I20250626 08:23:34.900949 1845 rpcz_store.cc:276] 0626 08:23:33.623822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:33.623941 (+ 119us) service_pool.cc:225] Handling call
0626 08:23:33.624318 (+ 377us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:33.949532 (+325214us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:34.900205 (+950673us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b540038ca54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:34.900388 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":634705792}
I20250626 08:23:34.912456 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:34.919597 1927 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:34.848837 3738 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.879029 3665 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.936090 3798 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:34.937083 3738 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.937311 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.937779 3738 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.846825 3741 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.938241 2582 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:34.939890 3718 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:34.952895 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 126306us
I20250626 08:23:34.942096 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:34.957325 1994 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:34.958160 2550 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:34.958541 3744 raft_consensus.cc:3053] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:34.958781 3738 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:34.959012 3744 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:34.959391 3738 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.941699 3790 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:34.942692 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.844739 3800 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.959277 1816 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:34.906894 3696 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:34.930169 3719 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:34.949100 1812 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:34.963090 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:34.972930 1927 raft_consensus.cc:2366] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:34.975394 2550 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:34.963908 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:34.958546 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:34.980713 2581 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:34.988117 3714 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:34.961454 3727 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:34.983120 2550 leader_election.cc:400] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:34.947090 3765 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.997890 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:34.981531 3781 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:34.999332 3781 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:34.999799 3781 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.000394 3696 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:34.980901 3817 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:34.952885 3667 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:34.982419 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.944448 3453 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.996929 2550 leader_election.cc:403] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20250626 08:23:35.006098 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.959896 3744 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:34.971546 3718 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:35.006785 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:23:35.018450 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:35.021247 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:35.037571 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:34.992446 3740 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:34.994858 1994 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:34.967317 3738 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.002281 2578 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:34.991654 3714 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.062183 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:35.072261 1923 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:35.038810 3718 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:35.063742 1921 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:35.061796 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:35.074954 1845 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.039201 3696 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.039438 3722 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:35.037382 3817 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:35.076746 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:34.987334 3719 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.078008 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:35.058187 3740 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:34.960032 3790 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.088399 1846 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:35.085121 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.093116 1921 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:35.096796 3719 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:35.075906 3718 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:35.090447 2579 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:35.078878 3826 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.077224 3744 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.085652 3817 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:35.105751 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.109062 1927 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:35.090399 3665 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:35.082101 3696 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.110586 1846 raft_consensus.cc:1273] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250626 08:23:35.119627 1843 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.042s sys 0.062s
W20250626 08:23:35.120045 1843 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.042s sys 0.062s
I20250626 08:23:35.122594 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.123324 1921 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:35.134045 1923 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:23:35.142781 1844 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1563) took 1781 ms (client timeout 1812 ms). Trace:
W20250626 08:23:35.143374 1844 rpcz_store.cc:269] 0626 08:23:33.344968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:33.345180 (+ 212us) service_pool.cc:225] Handling call
0626 08:23:35.126059 (+1780879us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b5400180454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:35.126202 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1709398272}
I20250626 08:23:35.150817 3837 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.111860 3665 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:35.113734 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:35.168587 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 77134.5us
I20250626 08:23:35.169236 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.169425 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:35.152796 1923 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:35.177430 3841 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.178081 3841 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.180500 3841 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.181138 3841 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:35.145334 3719 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:35.167881 3665 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.197932 1923 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:23:35.167999 3708 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.127725 3826 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.153945 3830 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.151391 3837 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.201867 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.268918 1923 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:35.202916 3790 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:35.206007 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 287191us
I20250626 08:23:35.205530 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:35.205951 2583 raft_consensus.cc:2391] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 2: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:35.191216 3821 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:23:35.269542 1845 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
W20250626 08:23:35.202328 3747 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.975s user 0.020s sys 0.027s
W20250626 08:23:35.270804 3834 log.cc:927] Time spent T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20250626 08:23:35.278240 2550 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:35.286821 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 117460us
I20250626 08:23:35.200166 3453 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.299677 1918 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:35.300506 3696 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.301079 3696 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.301697 3696 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.302773 3714 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.303735 3696 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.307749 3696 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.308296 3696 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.204610 3800 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:35.314035 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:35.273133 3678 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250626 08:23:35.317085 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:35.318034 1920 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:35.279033 3780 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.321768 3780 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.322952 3678 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 08:23:35.322419 3695 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.191928 3719 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:35.298517 3747 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s user 0.020s sys 0.027s
I20250626 08:23:35.333992 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.348392 1845 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:35.170668 3740 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.338316 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.323495 3821 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250626 08:23:35.351140 3718 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.353281 3718 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:35.355268 1844 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:35.211442 3708 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.383320 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:35.384610 1843 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:35.323843 3696 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.389364 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.391160 1845 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:35.355701 3678 tablet_bootstrap.cc:492] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Bootstrap complete.
I20250626 08:23:35.398747 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:35.400350 1814 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:35.401788 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 123666us
W20250626 08:23:35.374430 3677 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.275s user 0.022s sys 0.027s
I20250626 08:23:35.406747 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.373449 3669 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:35.373348 3695 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.411033 3844 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.407524 1846 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
W20250626 08:23:35.413123 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 224446us
I20250626 08:23:35.414932 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:35.315187 3703 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20250626 08:23:35.357244 3741 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.399405 3696 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:35.408331 3677 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.309s user 0.026s sys 0.028s
I20250626 08:23:35.276703 3830 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.373787 3738 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.425019 1967 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:35.415835 3669 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:35.379175 3847 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:35.446342 3846 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:35.446447 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.447065 3846 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.447597 3846 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.436797 3678 ts_tablet_manager.cc:1397] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent bootstrapping tablet: real 2.509s user 0.028s sys 0.019s
I20250626 08:23:35.466598 3848 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:35.467175 3848 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:35.467551 3848 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:35.470842 3837 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.426748 3790 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:35.437969 3841 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.441072 3669 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:35.440654 3738 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.423192 3774 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250626 08:23:35.490559 3826 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.500648 3826 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:35.501036 3852 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.501361 3826 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:35.504148 3708 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:35.504730 3708 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:35.493378 3719 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.487241 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:35.433483 3844 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.495719 3738 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.505350 3708 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.433065 3696 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.511196 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:35.533386 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:35.534193 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 111858us
W20250626 08:23:35.499329 3840 log.cc:927] Time spent T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20250626 08:23:35.526067 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:35.531241 2002 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:35.542698 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1880) took 1203 ms. Trace:
I20250626 08:23:35.484582 3847 raft_consensus.cc:695] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:35.455864 3718 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
W20250626 08:23:35.542593 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 259996us
I20250626 08:23:35.540637 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.545452 1918 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:35.546749 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.547825 1918 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 2: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:35.548936 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.550699 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.541872 2000 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:35.510022 3780 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.452931 3781 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.553122 1816 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:35.543226 2002 rpcz_store.cc:276] 0626 08:23:34.339734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.339854 (+ 120us) service_pool.cc:225] Handling call
0626 08:23:35.530681 (+1190827us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:35.530880 (+ 199us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.542410 (+ 11530us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:35.542643 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4462848}
I20250626 08:23:35.554003 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:35.560168 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:35.555739 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.513605 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:35.557849 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:35.561034 2002 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:35.484941 3841 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:35.543174 3668 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:35.562460 3693 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.555387 3719 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:35.650252 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.687899 1918 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:35.705565 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:35.707099 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:35.701884 3667 consensus_meta.cc:287] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.689s user 0.009s sys 0.009s
I20250626 08:23:35.729660 2001 raft_consensus.cc:1273] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:35.652874 3693 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:35.726182 1816 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:35.702831 3735 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.686249 1993 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:35.736539 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:35.607793 3780 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.117s
I20250626 08:23:35.744346 3781 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:35.724335 3738 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.736044 2001 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:35.654232 3668 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.501813 3852 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.654712 3849 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:35.748909 2001 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:35.704144 3719 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.754249 2001 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:35.754833 2000 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:35.735302 3844 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.763859 3861 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:35.741273 3667 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:35.742383 3856 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.653820 3826 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:35.788427 3665 consensus_meta.cc:287] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.583s user 0.010s sys 0.000s
I20250626 08:23:35.794142 1842 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:35.794638 3667 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:35.804332 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:35.820075 3667 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:35.750680 3738 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:35.835420 1812 consensus_peers.cc:487] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 -> Peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Couldn't send request to peer be56082cacb54e9aa8f1a9ee91e70d76. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:23:35.742169 3735 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.845245 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.813629 3719 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:35.859035 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 305121us
I20250626 08:23:35.835633 3702 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:35.846693 3735 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:35.845230 3825 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 1.211s user 0.004s sys 0.014s
I20250626 08:23:35.849527 3708 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:35.823292 3665 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.833320 3830 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.926718 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:35.897382 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 492266us
I20250626 08:23:35.764730 3861 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.925004 3853 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:35.927405 1846 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:35.934885 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.935619 1923 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:35.927886 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1881) took 1587 ms. Trace:
I20250626 08:23:35.763127 3849 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:35.856448 3696 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:35.863178 3702 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:35.938328 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:35.942727 1923 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.837793 3738 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.936936 1998 rpcz_store.cc:276] 0626 08:23:34.340014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.340120 (+ 106us) service_pool.cc:225] Handling call
0626 08:23:35.926964 (+1586844us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:35.927208 (+ 244us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.927427 (+ 219us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:35.927466 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:35.927542 (+ 76us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:35.927596 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:35.927668 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:35.927821 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4133248}
I20250626 08:23:35.948962 1850 raft_consensus.cc:1238] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:35.749365 3847 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:35.951009 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:35.959471 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:35.962961 1923 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:35.967571 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:35.957764 3731 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:35.974514 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:35.926075 3719 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.937316 3566 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.974666 3731 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:35.954800 3845 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:35.942965 3702 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:35.950695 3741 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.075603 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:35.947587 3830 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:35.979740 3815 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.213s user 0.007s sys 0.010s
I20250626 08:23:35.945343 3696 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:36.099689 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:36.110319 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:35.955916 3841 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:35.938879 3849 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:35.809818 3856 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:36.103161 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.29079e+06us
I20250626 08:23:36.103794 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:36.116396 2587 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:36.104722 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:36.118340 1965 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:36.119803 1814 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:36.120162 3845 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.121505 3790 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:36.122078 3790 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:36.112166 3830 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.016748 3695 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.098493 3759 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.112838 1995 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:36.075285 3708 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:35.981539 3784 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:36.099174 3566 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.106775 3837 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.136173 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 246) took 1953 ms. Trace:
I20250626 08:23:36.137262 3860 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:36.139532 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 248) took 1645 ms. Trace:
I20250626 08:23:36.151513 1997 rpcz_store.cc:276] 0626 08:23:34.494484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.494743 (+ 259us) service_pool.cc:225] Handling call
0626 08:23:34.511857 (+ 17114us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:34.512239 (+ 382us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:34.512283 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:34.555964 (+ 43681us) log.cc:844] Serialized 26 byte log entry
0626 08:23:34.630380 (+ 74416us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:34.630440 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:34.630535 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:35.817334 (+1186799us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:36.116755 (+299421us) raft_consensus.cc:1683] finished
0626 08:23:36.116787 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.139452 (+ 22665us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2818,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":71324928,"thread_start_us":33228,"threads_started":1,"wal-append.queue_time_us":40769}
I20250626 08:23:36.131371 3830 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.018144 3719 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.150627 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:36.143852 1995 rpcz_store.cc:276] 0626 08:23:34.182842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.183351 (+ 509us) service_pool.cc:225] Handling call
0626 08:23:36.107100 (+1923749us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:36.107300 (+ 200us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:36.135332 (+ 28032us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.135871 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17279872}
I20250626 08:23:36.169677 1846 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:36.017141 3851 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:36.167320 3669 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.657s user 0.013s sys 0.000s
I20250626 08:23:36.187830 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1891) took 1220 ms. Trace:
W20250626 08:23:36.139120 3817 consensus_meta.cc:287] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.016s user 0.013s sys 0.002s
I20250626 08:23:36.112807 3852 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:36.177378 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57809.6us
I20250626 08:23:36.125711 3668 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.188481 2002 rpcz_store.cc:276] 0626 08:23:34.967025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.648708 (+681683us) service_pool.cc:225] Handling call
0626 08:23:35.648979 (+ 271us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.651034 (+ 2055us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:23:35.729277 (+ 78243us) log.cc:844] Serialized 14 byte log entry
0626 08:23:36.187160 (+457883us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:36.187228 (+ 68us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:36.187298 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.187379 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.187763 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":95480,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":132581,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":3704,"thread_start_us":469481,"threads_started":2}
I20250626 08:23:36.189935 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.197085 3865 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.122967 3790 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.140504 3784 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:36.117952 3741 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.124011 3781 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20250626 08:23:36.187247 3665 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:36.238171 3848 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.770s user 0.007s sys 0.005s
W20250626 08:23:36.187935 3846 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.740s user 0.010s sys 0.004s
I20250626 08:23:36.188205 3860 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.128372 3696 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.142364 3759 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.237241 3738 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.147231 3708 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.262681 3853 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.140627 3695 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.259832 3852 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.168955 3837 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.217093 3729 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:36.290153 3738 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.271078 3784 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:36.283876 3812 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.629s user 0.003s sys 0.017s
I20250626 08:23:36.235184 3669 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.236171 3817 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.255398 3678 raft_consensus.cc:357] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:36.313943 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 331) took 2144 ms. Trace:
I20250626 08:23:36.339375 1926 rpcz_store.cc:276] 0626 08:23:34.155223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.165696 (+ 10473us) service_pool.cc:225] Handling call
0626 08:23:34.181181 (+ 15485us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:34.181645 (+ 464us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:34.181685 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:34.197471 (+ 15786us) log.cc:844] Serialized 26 byte log entry
0626 08:23:34.623970 (+426499us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:34.624033 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:34.624121 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:35.846094 (+1221973us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:36.282472 (+436378us) raft_consensus.cc:1683] finished
0626 08:23:36.282504 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.299932 (+ 17428us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9192,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":35696,"spinlock_wait_cycles":56915200,"thread_start_us":305164,"threads_started":1,"wal-append.queue_time_us":345851}
I20250626 08:23:36.315335 1844 raft_consensus.cc:2408] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250626 08:23:36.325999 3859 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20250626 08:23:36.327399 3738 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.281759 3860 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:36.363289 1921 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 08:23:36.366864 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1726) took 1236 ms. Trace:
I20250626 08:23:36.275182 3846 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.367296 1921 rpcz_store.cc:276] 0626 08:23:35.130325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.170189 (+ 39864us) service_pool.cc:225] Handling call
0626 08:23:36.362661 (+1192472us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:36.362902 (+ 241us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:36.366484 (+ 3582us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.366785 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6035328}
I20250626 08:23:36.329769 3566 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.335361 3744 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.262997 3851 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:36.401391 3851 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.404224 1918 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:36.278177 3848 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.363276 3861 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.385586 3731 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.384063 3719 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.383255 3696 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.344578 3678 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Initialized, Role: LEARNER
I20250626 08:23:36.389024 2583 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:36.385428 3708 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.313352 3865 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.406576 3744 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:36.426268 3693 consensus_meta.cc:287] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.680s user 0.011s sys 0.003s
I20250626 08:23:36.327442 3729 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.454655 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:36.323753 3852 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.449684 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1711) took 2220 ms. Trace:
I20250626 08:23:36.454026 3705 raft_consensus.cc:3053] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
W20250626 08:23:36.467520 1924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1676) took 1394 ms (client timeout 1539 ms). Trace:
I20250626 08:23:36.463752 1919 rpcz_store.cc:276] 0626 08:23:34.229019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.229197 (+ 178us) service_pool.cc:225] Handling call
0626 08:23:34.229666 (+ 469us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:34.230090 (+ 424us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:34.230117 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:34.294389 (+ 64272us) log.cc:844] Serialized 26 byte log entry
0626 08:23:34.585677 (+291288us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:34.585717 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:34.585806 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:36.255093 (+1669287us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:36.440401 (+185308us) raft_consensus.cc:1683] finished
0626 08:23:36.440433 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.449524 (+ 9091us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":481,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":59952,"spinlock_wait_cycles":77005184,"thread_start_us":290920,"threads_started":1,"wal-append.queue_time_us":331599}
W20250626 08:23:36.467972 1924 rpcz_store.cc:269] 0626 08:23:35.061317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.061430 (+ 113us) service_pool.cc:225] Handling call
0626 08:23:36.454477 (+1393047us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:23:36.456059 (+ 1582us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4504960}
I20250626 08:23:36.472457 3741 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:36.473541 3667 consensus_meta.cc:287] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.653s user 0.008s sys 0.004s
I20250626 08:23:36.474133 3667 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.476490 3667 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.468031 3693 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.479969 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39250 (request call id 1002) took 1045 ms. Trace:
I20250626 08:23:36.480937 1849 raft_consensus.cc:2366] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:36.467214 3705 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:36.481341 1927 rpcz_store.cc:276] 0626 08:23:35.434709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.435054 (+ 345us) service_pool.cc:225] Handling call
0626 08:23:36.479279 (+1044225us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54003124d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003a70 00007f4eb9087b1a 00007f4ebdfd509e 00007f4eb767f8c7 00007f4eb767f852 00007f4eb767f781 00007f4eb767f6fc 00007f4eb767e9b2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.479867 (+ 588us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562483072}
I20250626 08:23:36.493686 3693 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.494987 3705 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.496133 3705 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:36.504670 2583 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:36.502566 3695 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.439370 3845 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.509624 3744 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.510573 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 538) took 2009 ms. Trace:
I20250626 08:23:36.512286 2585 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:23:36.511114 2586 rpcz_store.cc:276] 0626 08:23:34.501236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.501377 (+ 141us) service_pool.cc:225] Handling call
0626 08:23:36.510322 (+2008945us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540060d1d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.510466 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559671168}
I20250626 08:23:36.421883 3851 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:36.522312 3824 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.887s user 0.005s sys 0.016s
I20250626 08:23:36.522910 3837 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.523739 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 330) took 2407 ms. Trace:
I20250626 08:23:36.524184 1928 rpcz_store.cc:276] 0626 08:23:34.116047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.116215 (+ 168us) service_pool.cc:225] Handling call
0626 08:23:34.116709 (+ 494us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:34.117184 (+ 475us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:34.117230 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:34.537167 (+419937us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b2c00046b18. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb7a226d1 00007f4eb7a31d02 00007f4eb90bac51 00007f4eb907f02a 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832
0626 08:23:34.537597 (+ 430us) log.cc:844] Serialized 26 byte log entry
0626 08:23:34.633813 (+ 96216us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:34.633861 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:34.633989 (+ 128us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:36.519507 (+1885518us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:36.523491 (+ 3984us) raft_consensus.cc:1683] finished
0626 08:23:36.523532 (+ 41us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.523672 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2203,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4697,"spinlock_wait_cycles":70838016,"thread_start_us":95743,"threads_started":1,"wal-append.queue_time_us":95486}
W20250626 08:23:36.526218 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 548) took 1557 ms (client timeout 1776 ms). Trace:
W20250626 08:23:36.526491 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:36.537292 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 476) took 1814 ms (client timeout 1514 ms). Trace:
I20250626 08:23:36.338629 3853 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:36.542188 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:36.526723 2579 rpcz_store.cc:269] 0626 08:23:34.968681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.971128 (+ 2447us) service_pool.cc:225] Handling call
0626 08:23:36.524711 (+1553583us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b5400045954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.526122 (+ 1411us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127584640}
W20250626 08:23:36.538009 2581 rpcz_store.cc:269] 0626 08:23:34.722734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.724047 (+ 1313us) service_pool.cc:225] Handling call
0626 08:23:36.536997 (+1812950us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54004362d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.537178 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325318400}
W20250626 08:23:36.552382 3820 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.499s user 0.016s sys 0.002s
I20250626 08:23:36.569193 3673 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.570106 3673 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:36.574067 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:36.561486 1846 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:23:36.575587 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:36.575999 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.576867 1846 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:36.580266 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.014s sys 0.038s
I20250626 08:23:36.539343 3669 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:36.582798 1843 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.019s sys 0.034s
W20250626 08:23:36.769052 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.216s user 0.014s sys 0.038s
W20250626 08:23:36.769333 1843 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s user 0.019s sys 0.034s
I20250626 08:23:36.771131 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.771617 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:36.489009 3678 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:36.584410 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.010s sys 0.040s
W20250626 08:23:36.772708 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.826s user 0.011s sys 0.041s
W20250626 08:23:36.773010 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 826346us
I20250626 08:23:36.773224 2002 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:36.571466 3738 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.597744 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 347) took 2009 ms. Trace:
I20250626 08:23:36.778081 1842 rpcz_store.cc:276] 0626 08:23:34.588502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.588654 (+ 152us) service_pool.cc:225] Handling call
0626 08:23:34.589025 (+ 371us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.810466 (+1221441us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.597396 (+786930us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540038c054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:36.597651 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2770412288}
I20250626 08:23:36.777370 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:36.779820 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.577879 3799 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.780596 2000 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:36.607614 3673 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.611147 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1593) took 1198 ms. Trace:
I20250626 08:23:36.580351 3781 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.786105 1845 rpcz_store.cc:276] 0626 08:23:35.412387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.413394 (+ 1007us) service_pool.cc:225] Handling call
0626 08:23:35.413935 (+ 541us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:35.556092 (+142157us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:35.556144 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:35.563759 (+ 7615us) log.cc:844] Serialized 26 byte log entry
0626 08:23:36.189682 (+625923us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:36.189741 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.189837 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:36.610813 (+420976us) raft_consensus.cc:1683] finished
0626 08:23:36.610850 (+ 37us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.611056 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":34924,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":598257,"threads_started":2,"wal-append.queue_time_us":945275}
I20250626 08:23:36.787180 3781 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.788558 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.594412 3846 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.635676 3865 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.638209 3817 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.789872 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.642468 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1718) took 1576 ms. Trace:
W20250626 08:23:36.644732 1848 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.041s sys 0.062s
I20250626 08:23:36.627427 3848 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:36.659934 1891 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.000s sys 0.002s
W20250626 08:23:36.791348 1848 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s user 0.041s sys 0.062s
W20250626 08:23:36.791522 1891 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.002s
I20250626 08:23:36.791221 1920 rpcz_store.cc:276] 0626 08:23:35.066117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.066712 (+ 595us) service_pool.cc:225] Handling call
0626 08:23:35.066960 (+ 248us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.324857 (+257897us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.642183 (+1317326us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54001e2154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:36.642396 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":560584960}
W20250626 08:23:36.670177 1892 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.025s sys 0.030s
I20250626 08:23:36.792608 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:36.792819 1892 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.025s sys 0.030s
I20250626 08:23:36.793807 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.794437 1848 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:36.679773 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.943s user 0.032s sys 0.044s
W20250626 08:23:36.795082 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.059s user 0.032s sys 0.044s
I20250626 08:23:36.795374 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:36.796347 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.797081 1848 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:36.797237 3763 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.797807 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.798240 3846 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.798835 3846 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.799168 3703 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.799945 3703 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.801402 3846 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:36.680781 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 102182us
I20250626 08:23:36.686856 3885 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:36.811659 3885 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.707937 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.812711 1850 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:36.812227 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:36.718390 1841 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 345) took 2186 ms (client timeout 2127 ms). Trace:
W20250626 08:23:36.815016 1841 rpcz_store.cc:269] 0626 08:23:34.528510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.528620 (+ 110us) service_pool.cc:225] Handling call
0626 08:23:36.714272 (+2185652us) spinlock_profiling.cc:243] Waited 85.9 ms on lock 0x7b54000343d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.714456 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197477120}
I20250626 08:23:36.710572 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.642575 3856 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:36.715338 3852 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:36.747076 2578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 545) took 2014 ms (client timeout 1560 ms). Trace:
W20250626 08:23:36.818045 2578 rpcz_store.cc:269] 0626 08:23:34.732615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.745475 (+ 12860us) service_pool.cc:225] Handling call
0626 08:23:36.745850 (+2000375us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54004024d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.746951 (+ 1101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":345901440}
I20250626 08:23:36.738687 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1681) took 1195 ms. Trace:
I20250626 08:23:36.819198 1918 rpcz_store.cc:276] 0626 08:23:35.543322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.706725 (+163403us) service_pool.cc:225] Handling call
0626 08:23:36.738479 (+1031754us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54001aa254. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.738615 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1600590208}
I20250626 08:23:36.819624 3703 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.739284 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1585) took 1692 ms. Trace:
I20250626 08:23:36.820427 1847 rpcz_store.cc:276] 0626 08:23:35.047298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.047504 (+ 206us) service_pool.cc:225] Handling call
0626 08:23:35.131066 (+ 83562us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:35.185739 (+ 54673us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:35.185783 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:35.185843 (+ 60us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:35.185886 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:35.186015 (+ 129us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:36.739087 (+1553072us) spinlock_profiling.cc:243] Waited 78.4 ms on lock 0x7b5400180450. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:36.739213 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33171,"raft.queue_time_us":79486,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":15628,"spinlock_wait_cycles":180344192,"thread_start_us":43710,"threads_started":1}
I20250626 08:23:36.728576 3668 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.822657 3668 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.825022 3668 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.731865 3666 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.825837 3666 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.828449 3666 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.828692 1845 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:36.754194 1742 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32866 (request call id 297) took 3300 ms. Trace:
I20250626 08:23:36.829237 1742 rpcz_store.cc:276] 0626 08:23:33.453847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:33.454006 (+ 159us) service_pool.cc:225] Handling call
0626 08:23:36.754065 (+3300059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:34.039043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:34.049413 (+ 10370us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:34.049445 (+ 32us) write_op.cc:435] Acquired schema lock
0626 08:23:34.049461 (+ 16us) tablet.cc:660] Decoding operations
0626 08:23:34.049810 (+ 349us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:34.049877 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:34.049932 (+ 55us) tablet.cc:683] Acquiring locks for 1 operations
0626 08:23:34.050127 (+ 195us) tablet.cc:699] Row locks acquired
0626 08:23:34.050146 (+ 19us) write_op.cc:260] PREPARE: finished
0626 08:23:34.050281 (+ 135us) write_op.cc:270] Start()
0626 08:23:34.050386 (+ 105us) write_op.cc:276] Timestamp: P: 1750926214050242 usec, L: 0
0626 08:23:34.050406 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 08:23:34.051025 (+ 619us) log.cc:844] Serialized 536 byte log entry
0626 08:23:34.328201 (+277176us) op_driver.cc:464] REPLICATION: finished
0626 08:23:34.551401 (+223200us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b1800103600. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611d555 00007f4eb9013660 00007f4eb900d017 00007f4eb900c309
0626 08:23:34.742622 (+191221us) write_op.cc:301] APPLY: starting
0626 08:23:34.749471 (+ 6849us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:34.786966 (+ 37495us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:34.789945 (+ 2979us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:34.831784 (+ 41839us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:34.866811 (+ 35027us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 08:23:34.886637 (+ 19826us) write_op.cc:312] APPLY: finished
0626 08:23:34.943774 (+ 57137us) log.cc:844] Serialized 23 byte log entry
0626 08:23:34.953575 (+ 9801us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:34.964840 (+ 11265us) write_op.cc:454] Released schema lock
0626 08:23:34.966222 (+ 1382us) write_op.cc:341] FINISH: Updating metrics
0626 08:23:34.997672 (+ 31450us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:34.997706 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2061,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":383943,"apply.run_cpu_time_us":3271,"apply.run_wall_time_us":252464,"num_ops":1,"prepare.queue_time_us":194606,"prepare.run_cpu_time_us":13663,"prepare.run_wall_time_us":111345,"raft.queue_time_us":65076,"raft.run_cpu_time_us":22284,"raft.run_wall_time_us":513884,"replication_time_us":245564,"spinlock_wait_cycles":67529472,"thread_start_us":563671,"threads_started":4,"wal-append.queue_time_us":73133,"wal-append.run_cpu_time_us":20471,"wal-append.run_wall_time_us":1851552}]]}
I20250626 08:23:35.930604 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:36.840158 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:36.841053 1919 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:36.847101 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:36.847872 1919 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 4
W20250626 08:23:36.723644 1923 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.006s sys 0.003s
I20250626 08:23:36.772384 1995 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:36.525851 3851 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.780963 3799 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.856207 3851 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.856388 3799 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.782501 3873 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.857312 3676 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.857770 3873 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.858253 3676 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.784054 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:36.772396 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.631284 3841 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:36.864907 3849 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.737s user 0.013s sys 0.000s
I20250626 08:23:36.802296 3881 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.804890 3673 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:36.805130 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:36.806434 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:36.867419 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.091s sys 0.187s
I20250626 08:23:36.867801 1927 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:36.867970 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.091s sys 0.187s
W20250626 08:23:36.868345 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 667482us
I20250626 08:23:36.797761 3763 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.791358 3781 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:36.871943 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:36.810474 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:36.813984 3885 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:36.752571 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:36.754307 3853 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:36.761585 1996 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.005s sys 0.005s
W20250626 08:23:36.923475 1996 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.006s sys 0.006s
I20250626 08:23:36.860545 3682 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:36.862691 3872 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:36.932581 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.005s sys 0.014s
W20250626 08:23:36.933112 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.005s sys 0.014s
I20250626 08:23:36.864522 3841 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:36.680111 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.010s sys 0.064s
W20250626 08:23:36.937625 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.981s user 0.010s sys 0.064s
I20250626 08:23:36.807241 1816 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:36.891635 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:36.808517 3848 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:36.943046 1928 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 4
W20250626 08:23:36.943163 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 136001us
I20250626 08:23:36.883998 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:36.929679 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:36.952593 1816 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:36.956878 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:36.964501 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:36.865427 3849 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:36.936801 3703 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:36.951800 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:36.921303 2583 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:36.922680 1919 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:36.879525 1965 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.196:37733 timed out after 1.514s (SENT) [suppressed 42 similar messages]
I20250626 08:23:36.922253 3673 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:36.965966 3781 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:36.948500 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:36.946449 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 62599.4us
I20250626 08:23:36.976451 3681 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:36.978142 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:36.949707 3885 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:36.971817 3857 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250626 08:23:37.001649 3869 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:36.993697 3790 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:36.970928 3799 consensus_queue.cc:1046] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20250626 08:23:37.003373 3868 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:37.003672 3784 consensus_meta.cc:287] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.667s user 0.011s sys 0.002s
I20250626 08:23:36.865679 3881 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:37.006733 3702 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.894s user 0.012s sys 0.000s
I20250626 08:23:36.947436 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:37.054934 1927 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:37.055171 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:36.960459 3722 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:36.944844 3846 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:37.022189 1848 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:36.959919 3800 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:37.022337 1928 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:37.039539 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 67817.4us
I20250626 08:23:37.024259 3817 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:37.008316 3886 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:37.050148 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:37.055822 2000 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:37.057070 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.078058 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.078176 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.078684 2002 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:37.079671 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.078684 2000 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:37.083860 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.084575 2000 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:36.887346 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:36.871222 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:36.872911 1997 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:37.042095 3673 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:37.071162 2001 raft_consensus.cc:1273] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 08:23:37.023567 3885 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:36.960217 3872 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.015626 3703 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.100831 1918 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:37.072909 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 138838us
I20250626 08:23:37.075672 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:37.043175 3790 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:37.086028 2585 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:37.096860 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:37.059376 3866 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:37.097383 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:37.096376 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.013s sys 0.017s
I20250626 08:23:37.108405 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:37.108500 1995 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:37.109122 1814 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:37.109530 1890 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:37.091472 3860 consensus_meta.cc:287] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.693s user 0.013s sys 0.000s
I20250626 08:23:37.058584 3722 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:37.118497 2000 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:37.124577 3873 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:37.122357 3818 raft_consensus.cc:2762] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250626 08:23:37.132601 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 252) took 2560 ms. Trace:
I20250626 08:23:37.133096 1999 rpcz_store.cc:276] 0626 08:23:34.571812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.572234 (+ 422us) service_pool.cc:225] Handling call
0626 08:23:36.460065 (+1887831us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:36.460307 (+ 242us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:36.460741 (+ 434us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:36.460770 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:36.467031 (+ 6261us) log.cc:844] Serialized 26 byte log entry
0626 08:23:36.958309 (+491278us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:36.958362 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.958609 (+ 247us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:37.132274 (+173665us) raft_consensus.cc:1683] finished
0626 08:23:37.132295 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:37.132506 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":9479,"spinlock_wait_cycles":37472128,"thread_start_us":490797,"threads_started":1,"wal-append.queue_time_us":663293}
W20250626 08:23:37.136548 3878 log.cc:927] Time spent T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20250626 08:23:37.087898 3868 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.140169 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:37.140542 3872 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:37.141829 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:37.139381 3666 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:37.106956 3886 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:37.143589 3886 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:37.144331 3881 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:37.070313 3738 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:37.060514 3781 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:37.064539 3869 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.142199 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:37.142342 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 255) took 2492 ms. Trace:
I20250626 08:23:37.070152 3851 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:37.119024 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.936s user 0.013s sys 0.017s
I20250626 08:23:37.060906 3702 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:37.075577 3800 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.088977 3817 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.063376 3784 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:37.076890 3846 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.226339 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:38.079710 3877 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.903s user 0.004s sys 0.020s
W20250626 08:23:38.230752 3877 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s user 0.006s sys 0.020s
I20250626 08:23:38.232796 3668 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:37.171072 3849 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.234987 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:38.155083 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1732) took 2223 ms. Trace:
I20250626 08:23:38.236394 3851 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.486357 3774 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.238399 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.236864 1923 rpcz_store.cc:276] 0626 08:23:35.931489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.126681 (+195192us) service_pool.cc:225] Handling call
0626 08:23:36.132715 (+ 6034us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:36.133274 (+ 559us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:36.133309 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:36.142369 (+ 9060us) log.cc:844] Serialized 26 byte log entry
0626 08:23:36.857079 (+714710us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:36.857143 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.859814 (+ 2671us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:37.162217 (+302403us) raft_consensus.cc:1683] finished
0626 08:23:37.162247 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.154650 (+992403us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125,"raft.queue_time_us":47599,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":48392,"thread_start_us":635459,"threads_started":1,"wal-append.queue_time_us":697215}
I20250626 08:23:38.239042 3774 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.239925 3851 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.240940 2581 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 LEARNER]: Advancing to term 2
I20250626 08:23:38.236527 2579 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:38.242508 3738 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.160403 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.245348 1999 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:37.108760 3885 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.236744 3781 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:37.181774 1994 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:37.678911 1925 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.030s sys 0.078s
W20250626 08:23:37.656216 1965 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.010s sys 0.013s
W20250626 08:23:37.517158 1893 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.014s sys 0.031s
I20250626 08:23:38.245662 3738 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:37.146782 3722 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:37.252521 3743 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.008s sys 0.023s
W20250626 08:23:38.272527 3743 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.629s user 0.011s sys 0.033s
I20250626 08:23:38.274433 3738 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.275156 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 265) took 1153 ms. Trace:
I20250626 08:23:38.260902 3668 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.282140 3759 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.282465 3763 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:38.261446 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 1.12071e+06us
I20250626 08:23:38.283274 3668 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.283959 1814 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:38.285735 3759 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:38.286392 3759 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:37.245548 3714 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.287866 3714 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.289649 1920 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:38.287909 3759 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.292155 3846 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.271512 3868 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:38.264017 1996 rpcz_store.cc:276] 0626 08:23:34.649905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.838637 (+188732us) service_pool.cc:225] Handling call
0626 08:23:34.839010 (+ 373us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:34.839410 (+ 400us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:34.839451 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:34.840002 (+ 551us) log.cc:844] Serialized 26 byte log entry
0626 08:23:36.067294 (+1227292us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b180033aa20. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc2f01 00007f4eb9051ccf
0626 08:23:36.923811 (+856517us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:36.923840 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:36.923889 (+ 49us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:37.141940 (+218051us) raft_consensus.cc:1683] finished
0626 08:23:37.141978 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:37.142266 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":859,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":19896,"spinlock_wait_cycles":41971456,"thread_start_us":856327,"threads_started":1,"wal-append.queue_time_us":2273732,"wal-append.run_cpu_time_us":2102,"wal-append.run_wall_time_us":1111715}
I20250626 08:23:37.183094 3666 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:38.267903 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1742) took 1415 ms. Trace:
W20250626 08:23:38.270044 1995 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.006s sys 0.005s
I20250626 08:23:38.270402 3849 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250626 08:23:37.487440 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 455269us
I20250626 08:23:38.271019 3774 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.261111 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.279793 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1887) took 3362 ms. Trace:
I20250626 08:23:38.279994 3817 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.292212 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1735) took 1497 ms. Trace:
I20250626 08:23:37.171413 3678 ts_tablet_manager.cc:1428] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent starting tablet: real 1.692s user 0.019s sys 0.010s
I20250626 08:23:38.275835 2000 rpcz_store.cc:276] 0626 08:23:37.262606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:38.223234 (+960628us) service_pool.cc:225] Handling call
0626 08:23:38.273927 (+ 50693us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:38.274400 (+ 473us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:38.274444 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:38.274518 (+ 74us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:38.274605 (+ 87us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.274685 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.274892 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:37.899643 3886 consensus_meta.cc:287] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.755s user 0.007s sys 0.005s
W20250626 08:23:37.895794 3681 consensus_meta.cc:287] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.872s user 0.008s sys 0.004s
W20250626 08:23:37.657186 3847 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.338s user 0.009s sys 0.034s
W20250626 08:23:37.497905 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:38.233016 3821 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:37.122668 3790 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.295656 1918 rpcz_store.cc:276] 0626 08:23:36.852702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.886981 (+ 34279us) service_pool.cc:225] Handling call
0626 08:23:38.267374 (+1380393us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5400301454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:38.267663 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32523520}
I20250626 08:23:37.183985 3841 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.297278 3849 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.318940 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 264) took 1249 ms. Trace:
I20250626 08:23:38.319350 3849 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.319423 1997 rpcz_store.cc:276] 0626 08:23:37.068980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.127996 (+ 59016us) service_pool.cc:225] Handling call
0626 08:23:37.437661 (+309665us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:38.241155 (+803494us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:38.241224 (+ 69us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:38.241292 (+ 68us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:38.241345 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.318485 (+ 77140us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b54001315d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:38.318529 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.318740 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1846957312}
I20250626 08:23:38.321323 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:38.302209 1995 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.173s user 0.006s sys 0.005s
W20250626 08:23:38.314394 1893 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.331s user 0.018s sys 0.042s
W20250626 08:23:38.323094 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.35647e+06us
I20250626 08:23:38.324411 3784 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.326013 3702 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.301714 3869 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.304996 1920 rpcz_store.cc:276] 0626 08:23:36.794745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.810141 (+ 15396us) service_pool.cc:225] Handling call
0626 08:23:38.291979 (+1481838us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400312ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:38.292147 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3147376000}
I20250626 08:23:37.141835 3860 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.327281 3821 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.329057 3860 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.303321 1994 rpcz_store.cc:276] 0626 08:23:34.917063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.200996 (+283933us) service_pool.cc:225] Handling call
0626 08:23:37.159502 (+1958506us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:38.255676 (+1096174us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.279444 (+ 23768us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54002035d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:38.279625 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4460504320}
I20250626 08:23:38.330135 3800 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:38.314338 3666 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.330889 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:38.331275 1997 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:38.331996 1994 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.332535 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:38.333334 1997 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:37.146586 3866 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.345064 3866 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.326356 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.346104 1996 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:38.346290 3886 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.347497 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.348282 1996 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:38.348953 3886 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.349246 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:38.333973 1965 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.238s user 0.011s sys 0.015s
W20250626 08:23:38.334318 1925 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.580s user 0.030s sys 0.081s
I20250626 08:23:38.334470 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1612) took 1199 ms. Trace:
I20250626 08:23:38.335131 1842 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:37.179464 3703 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:38.336805 1893 connection.cc:232] RPC call timeout handler was delayed by -1.132691412999975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
W20250626 08:23:38.337872 3678 ts_tablet_manager.cc:1460] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Tablet startup took 2842ms
I20250626 08:23:38.338873 1999 raft_consensus.cc:2391] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:38.352461 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1304779980000603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
W20250626 08:23:38.354295 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1304779980000603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20250626 08:23:38.352008 3844 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.350956 1845 rpcz_store.cc:276] 0626 08:23:37.135320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.141469 (+ 6149us) service_pool.cc:225] Handling call
0626 08:23:38.334227 (+1192758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:38.340957 3790 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.275525 3738 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.352662 3702 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.359593 1996 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:38.357185 3886 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.312402 3743 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20250626 08:23:38.374861 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:38.378115 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.49851e+06us
I20250626 08:23:38.368611 3821 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.360743 3844 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.365893 3817 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:38.389303 1843 raft_consensus.cc:2366] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:38.385535 3886 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.396860 1999 raft_consensus.cc:2408] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250626 08:23:38.399123 3790 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.403033 3729 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.403512 3886 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.403728 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.404951 2001 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:38.405716 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 263) took 1337 ms. Trace:
I20250626 08:23:38.407647 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.408106 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.408533 1928 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:38.408910 1921 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:38.408118 3702 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.412707 3702 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.413444 3702 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:38.393265 3847 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 2.023s user 0.010s sys 0.037s
I20250626 08:23:38.422600 3702 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.416595 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 558) took 1295 ms. Trace:
I20250626 08:23:38.408124 3791 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:23:38.426927 1893 leader_election.cc:400] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:38.427386 1893 leader_election.cc:403] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:23:38.427292 3791 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:38.428817 3784 consensus_queue.cc:1046] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:38.429235 3841 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:38.406701 2001 rpcz_store.cc:276] 0626 08:23:37.068682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.092529 (+ 23847us) service_pool.cc:225] Handling call
0626 08:23:38.403254 (+1310725us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:23:38.405612 (+ 2358us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17755392}
I20250626 08:23:38.425714 3702 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.435020 3579 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.442229 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.445324 2001 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
I20250626 08:23:38.444897 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:38.448683 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:38.448181 2551 connection.cc:232] RPC call timeout handler was delayed by -1.0174704709997968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20250626 08:23:38.457249 1920 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:38.435796 3847 raft_consensus.cc:3053] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:38.460429 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.463289 1918 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:38.442265 3579 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:38.467909 3703 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.471696 3738 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.471592 3703 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.474953 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.443861 3817 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.459453 3847 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:38.490558 1999 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:38.478137 1968 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:38.473867 3703 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:38.518718 1847 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.344s user 0.063s sys 0.090s
I20250626 08:23:38.519450 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:38.522120 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 59093.3us
I20250626 08:23:38.523180 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.426107 2578 rpcz_store.cc:276] 0626 08:23:37.120811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.121051 (+ 240us) service_pool.cc:225] Handling call
0626 08:23:37.130185 (+ 9134us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:38.278530 (+1148345us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:38.278594 (+ 64us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:38.278667 (+ 73us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:38.278740 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.415995 (+137255us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003eec54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:38.416070 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.416347 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2639650688}
W20250626 08:23:38.523648 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 140553us
I20250626 08:23:38.523996 3791 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.525134 1965 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:38.530308 1847 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.393s user 0.064s sys 0.090s
I20250626 08:23:38.520967 3847 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.478101 3861 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.534153 3670 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:38.535112 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:38.541700 1997 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:38.546866 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.547890 1844 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:38.535909 3738 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:38.551096 3738 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:38.551400 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.551590 3738 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:38.552117 3738 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.552659 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.554314 3791 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:38.553778 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.532018 3860 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.566481 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:38.568110 1997 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:38.574460 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.575242 1999 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:38.531186 3722 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.594406 1997 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:23:38.353384 3678 ts_tablet_manager.cc:1462] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Trace:
0626 08:23:29.869676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:29.881012 (+ 11336us) service_pool.cc:225] Handling call
0626 08:23:29.966081 (+ 85069us) ts_tablet_manager.cc:928] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Initiating tablet copy from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
0626 08:23:30.941753 (+975672us) tablet_copy_client.cc:368] Tablet copy session begun
0626 08:23:32.033104 (+1091351us) tablet_metadata.cc:641] Metadata flushed
0626 08:23:32.033389 (+ 285us) tablet_copy_client.cc:497] Wrote new tablet metadata
0626 08:23:32.270649 (+237260us) tablet_copy_client.cc:418] Wrote new consensus metadata
0626 08:23:32.270949 (+ 300us) tablet_replica.cc:206] Creating consensus instance
0626 08:23:32.271298 (+ 349us) ts_tablet_manager.cc:983] Replica registered.
0626 08:23:32.271358 (+ 60us) inbound_call.cc:173] Queueing success response
0626 08:23:32.925015 (+653657us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0626 08:23:35.487894 (+2562879us) ts_tablet_manager.cc:1429] Starting tablet replica
0626 08:23:35.754445 (+266551us) tablet_replica.cc:260] Starting instrumentation
0626 08:23:35.755104 (+ 659us) tablet_replica.cc:279] Starting consensus
Metrics: {"dns_us":62367,"mutex_wait_us":182445,"raft.queue_time_us":3297,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":26737,"tablet-copy.queue_time_us":83813,"tablet-download-pool.queue_time_us":183,"tablet-download-pool.run_cpu_time_us":1566,"tablet-download-pool.run_wall_time_us":40949,"thread_start_us":142247,"threads_started":2}
I20250626 08:23:38.601130 1920 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:38.601747 2001 raft_consensus.cc:2391] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:38.544808 3851 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.606467 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.608819 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.616096 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:38.629665 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 114299us
I20250626 08:23:38.628545 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.596096 3821 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.623157 3847 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.633340 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:38.592185 3860 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.636193 1918 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:38.640960 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 259) took 1573 ms. Trace:
I20250626 08:23:38.640748 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:38.644120 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:38.641733 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 106727us
I20250626 08:23:38.646276 1997 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:38.645190 2001 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:38.654039 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46433us
I20250626 08:23:38.655895 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.656299 3857 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.656916 1849 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:38.657116 3857 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.659274 3722 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.659652 3857 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.660301 3817 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:38.661015 3895 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250626 08:23:38.662359 3860 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:38.662653 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1739) took 1856 ms. Trace:
I20250626 08:23:38.663038 3860 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:38.663533 3860 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.664189 3860 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.663125 1924 rpcz_store.cc:276] 0626 08:23:36.806400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.806525 (+ 125us) service_pool.cc:225] Handling call
0626 08:23:36.806861 (+ 336us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:36.807192 (+ 331us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:36.807219 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:36.807737 (+ 518us) log.cc:844] Serialized 26 byte log entry
0626 08:23:37.537867 (+730130us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:37.537952 (+ 85us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:37.538057 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:38.662246 (+1124189us) raft_consensus.cc:1683] finished
0626 08:23:38.662282 (+ 36us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.662479 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2370,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":166,"thread_start_us":492757,"threads_started":1,"wal-append.queue_time_us":1654626}
I20250626 08:23:38.666491 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.667519 3747 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.667950 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.668335 3747 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.589874 3861 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.669643 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.635967 3866 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.638296 3743 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.643654 3703 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:38.676939 3703 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.678409 3849 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.679118 3905 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:23:38.679937 1850 raft_consensus.cc:1238] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:38.642519 1995 rpcz_store.cc:276] 0626 08:23:37.067540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.118689 (+ 51149us) service_pool.cc:225] Handling call
0626 08:23:37.128645 (+ 9956us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:37.128860 (+ 215us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:38.323096 (+1194236us) log.cc:844] Serialized 14 byte log entry
0626 08:23:38.601810 (+278714us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:38.609945 (+ 8135us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:38.620070 (+ 10125us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.624994 (+ 4924us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.640900 (+ 15906us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1318482,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":1453204,"threads_started":2}
I20250626 08:23:38.689145 3666 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.689939 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.690759 2000 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:38.691699 3914 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:38.693259 2000 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:38.694352 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1716) took 3629 ms. Trace:
I20250626 08:23:38.694523 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.695566 1849 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:38.695647 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.696005 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.696137 3579 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.679322 3703 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.680863 3849 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.715113 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.717830 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.721187 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.698143 3844 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.728780 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.699277 1925 rpcz_store.cc:276] 0626 08:23:35.064787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:35.064896 (+ 109us) service_pool.cc:225] Handling call
0626 08:23:35.065146 (+ 250us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:35.083032 (+ 17886us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:35.083081 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:36.651000 (+1567919us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b2c00046b18. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb7a226d1 00007f4eb7a31d02 00007f4eb90bac51 00007f4eb907f02a 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832
0626 08:23:36.710124 (+ 59124us) log.cc:844] Serialized 26 byte log entry
0626 08:23:38.402070 (+1691946us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:38.402135 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.402259 (+ 124us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:38.681345 (+279086us) raft_consensus.cc:1683] finished
0626 08:23:38.681371 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.694264 (+ 12893us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17535,"raft.queue_time_us":25227,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":5453,"spinlock_wait_cycles":6026624,"thread_start_us":1628535,"threads_started":1,"wal-append.queue_time_us":1773118}
W20250626 08:23:38.729359 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 53746.8us
I20250626 08:23:38.671662 3851 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.735563 1928 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:38.703042 3821 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.743655 1921 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:38.754022 1920 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:38.756521 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:38.756444 3885 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.759121 3885 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:38.761689 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 131094us
I20250626 08:23:38.762513 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.764297 1999 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 1.
W20250626 08:23:38.765923 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:38.744491 3759 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.727824 3866 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.772804 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:38.773748 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.774715 3866 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.774998 1919 raft_consensus.cc:2408] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250626 08:23:38.775537 1842 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1578) took 2656 ms (client timeout 2225 ms). Trace:
I20250626 08:23:38.775956 1928 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:38.776183 1842 rpcz_store.cc:269] 0626 08:23:36.119504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.779488 (+659984us) service_pool.cc:225] Handling call
0626 08:23:38.775221 (+1995733us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400107a54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:38.775421 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3574540544}
I20250626 08:23:38.773630 1927 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.770054 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1613) took 1631 ms. Trace:
I20250626 08:23:38.760957 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:38.727624 3743 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.783917 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:38.784598 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:38.785312 3910 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250626 08:23:38.795311 1927 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:38.796518 1741 catalog_manager.cc:5582] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 3 to 4, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:23:38.802280 3861 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.802589 3676 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.803706 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:38.805220 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 85065.1us
I20250626 08:23:38.808130 3800 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:38.808308 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:38.808620 3800 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:38.809026 3800 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:38.748267 3666 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.800364 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1594) took 1739 ms. Trace:
I20250626 08:23:38.800832 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.778635 1847 rpcz_store.cc:276] 0626 08:23:37.135659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.135845 (+ 186us) service_pool.cc:225] Handling call
0626 08:23:37.136314 (+ 469us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:37.136555 (+ 241us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:37.136585 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:37.137174 (+ 589us) log.cc:844] Serialized 26 byte log entry
0626 08:23:38.533643 (+1396469us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:38.533745 (+ 102us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.533851 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:38.761561 (+227710us) raft_consensus.cc:1683] finished
0626 08:23:38.761576 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.766674 (+ 5098us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1393023,"threads_started":1,"wal-append.queue_time_us":1488113}
I20250626 08:23:38.798065 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.810434 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.753021 3844 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.817638 2002 rpcz_store.cc:276] 0626 08:23:37.073498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.093259 (+ 19761us) service_pool.cc:225] Handling call
0626 08:23:38.799233 (+1705974us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:38.799544 (+ 311us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:38.799790 (+ 246us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:38.799829 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:38.799898 (+ 69us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:38.799951 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.800019 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.800157 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16260352}
I20250626 08:23:38.824628 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:38.831418 3759 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.833876 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:38.834679 1999 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.835522 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.836341 2583 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:38.828042 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:38.830139 3849 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.839018 3849 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.841140 3849 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.837069 3666 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:38.841881 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 70685.7us
I20250626 08:23:38.843730 1996 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:38.853895 1967 connection.cc:232] RPC call timeout handler was delayed by -2.2267720729996654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
W20250626 08:23:38.854547 1967 connection.cc:232] RPC call timeout handler was delayed by -1.5397466249996796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20250626 08:23:38.854911 1967 connection.cc:232] RPC call timeout handler was delayed by -1.15364662499951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20250626 08:23:38.855270 1967 connection.cc:232] RPC call timeout handler was delayed by -1.0231466249997538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20250626 08:23:38.855674 1967 connection.cc:232] RPC call timeout handler was delayed by -1.0006466249997175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
I20250626 08:23:38.855459 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.856524 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.826830 3885 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:38.816079 3886 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:38.861789 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 90453.5us
I20250626 08:23:38.860642 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:38.865144 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.865965 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.865041 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.867094 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.873709 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:38.874678 1926 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.851708 3747 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.855494 3821 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.886448 1919 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250626 08:23:38.889977 1925 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:38.890913 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 141165us
I20250626 08:23:38.891144 1923 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:38.892287 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.860420 3861 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:38.858520 3914 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:38.895006 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 70619.1us
I20250626 08:23:38.901877 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.902423 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.901928 1996 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:38.910933 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:38.923381 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:38.905759 1968 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:38.927676 3914 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:38.928508 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:38.930050 3918 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.930857 3918 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.932363 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1584) took 2160 ms. Trace:
I20250626 08:23:38.873973 3844 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.934132 3920 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:38.878319 3885 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.903421 3722 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:38.941324 2002 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:38.957780 3813 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:38.958356 3851 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.958451 3813 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:38.959115 3813 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:38.960191 1816 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:38.960479 1994 consensus_meta.cc:287] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.627s user 0.013s sys 0.001s
I20250626 08:23:38.961124 1994 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:38.963191 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:38.964026 3844 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:38.965122 3844 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:38.966007 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:38.966189 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:38.967221 3861 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:38.968053 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 94853.8us
I20250626 08:23:38.967882 3861 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:38.954416 3722 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.974833 3722 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:38.975939 1843 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1611) took 1841 ms (client timeout 1522 ms). Trace:
W20250626 08:23:38.976387 1843 rpcz_store.cc:269] 0626 08:23:37.134960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:37.139909 (+ 4949us) service_pool.cc:225] Handling call
0626 08:23:38.975682 (+1835773us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540031f1d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:38.975859 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2812676480}
I20250626 08:23:38.978364 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:38.933610 3918 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.979194 3759 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:38.979677 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.965620 3844 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:38.987471 1999 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:38.992514 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:38.989356 1925 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:38.969656 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:38.980230 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:38.900547 3860 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:38.932891 1998 rpcz_store.cc:276] 0626 08:23:36.771973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.772644 (+ 671us) service_pool.cc:225] Handling call
0626 08:23:36.772961 (+ 317us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:36.860971 (+ 88010us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:36.861002 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:36.866222 (+ 5220us) log.cc:844] Serialized 26 byte log entry
0626 08:23:38.671154 (+1804932us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b18001ba4e0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc2f01 00007f4eb9051ccf
0626 08:23:38.781240 (+110086us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:38.816879 (+ 35639us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:38.844669 (+ 27790us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:38.931846 (+ 87177us) raft_consensus.cc:1683] finished
0626 08:23:38.931870 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:38.932144 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97301,"raft.queue_time_us":87734,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":60411,"spinlock_wait_cycles":17365888,"thread_start_us":71648,"threads_started":1,"wal-append.queue_time_us":1941895}
W20250626 08:23:38.994189 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 171490us
I20250626 08:23:38.905009 3886 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:38.936597 3920 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:38.990900 3844 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:38.994233 1816 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:38.998314 3738 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:39.008095 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.003791 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.000697 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.015604 3759 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.016093 3759 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:39.016551 3759 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.019860 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.020776 1920 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:38.993175 3851 raft_consensus.cc:2762] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250626 08:23:39.021762 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.022033 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:38.999300 3817 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:39.022604 1928 raft_consensus.cc:2366] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3.
I20250626 08:23:39.027112 3885 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.028291 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.013701 1845 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:39.014243 1850 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:38.993405 3914 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:39.035197 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.036128 1847 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:23:39.037284 1926 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:39.038166 1841 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.014557 2002 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:38.999639 3908 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:39.044445 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.045011 1996 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:39.045418 1925 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:39.006506 3920 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.043650 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.013738 3861 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:39.007472 3918 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:39.049083 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:39.055910 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:39.061353 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.063841 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.064903 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:39.066144 1848 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1579) took 2892 ms (client timeout 3169 ms). Trace:
W20250626 08:23:39.066596 1848 rpcz_store.cc:269] 0626 08:23:36.120453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:36.797567 (+677114us) service_pool.cc:225] Handling call
0626 08:23:38.999334 (+2201767us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400211954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:39.013051 (+ 13717us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":513950976}
I20250626 08:23:39.067032 3908 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.069422 3908 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:39.071086 3714 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.022614 3817 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:39.073678 1841 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:39.076596 1928 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250626 08:23:39.077997 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 108560us
I20250626 08:23:39.084862 2000 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.085887 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.083873 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.088106 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.088810 1921 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:39.096002 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.075495 3817 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:39.096902 1848 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:39.091707 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:39.047036 3722 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.098958 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:39.099682 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 72119.6us
I20250626 08:23:39.075474 3743 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.106719 2586 raft_consensus.cc:2391] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:23:39.106787 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 215085us
I20250626 08:23:39.055197 3866 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250626 08:23:39.085517 3670 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.099323 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:39.126185 1814 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:39.109218 3908 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.127238 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.127688 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.131683 2582 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.115381 3918 raft_consensus.cc:2762] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250626 08:23:39.094406 3714 consensus_queue.cc:1046] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250626 08:23:39.139767 1846 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:39.125766 1965 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:39.143054 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 51385.5us
I20250626 08:23:39.152277 1814 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:39.115041 3847 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250626 08:23:39.161511 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.164311 1967 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.129679 3886 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.167752 1816 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.170029 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.170167 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.171195 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:39.151726 3849 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.120425 3670 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.177439 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.152810 3860 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.178973 3715 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.201578 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:39.164332 3738 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:39.227551 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 48259.8us
I20250626 08:23:39.240931 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:39.175976 3774 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.239529 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.252596 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.177897 3849 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:39.202307 3886 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.160073 3908 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.262336 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:39.268409 3784 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:39.270347 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 117293us
I20250626 08:23:39.271327 1816 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:39.271519 1814 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:39.248097 3774 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:39.272329 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:39.272763 3800 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.273758 3848 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:39.272358 3774 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.274241 3848 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:39.274374 2581 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:39.274725 3848 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:39.276518 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.277045 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.277282 1846 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:23:39.278327 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:39.278743 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 153056us
I20250626 08:23:39.279251 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.279616 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.279888 1850 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:23:39.280448 1842 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:39.239404 3918 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.281997 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:39.282387 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.283528 1938 tablet_copy_service.cc:342] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Request end of tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-ac6c556e015045ac949f4ee3d2048b56 received from {username='slave'} at 127.0.0.1:58462
I20250626 08:23:39.283864 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:39.284047 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.284261 1938 tablet_copy_service.cc:434] P 74a4b66e1f7b4aed8fae26d31f9cc4e7: ending tablet copy session be56082cacb54e9aa8f1a9ee91e70d76-ac6c556e015045ac949f4ee3d2048b56 on tablet ac6c556e015045ac949f4ee3d2048b56 with peer be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:39.284787 1920 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:39.285169 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.285679 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.282018 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.286217 3791 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:39.286355 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.286775 1843 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:39.286871 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.278375 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:39.262820 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 146269us
I20250626 08:23:39.287607 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:39.297513 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 136981us
I20250626 08:23:39.269704 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:39.301170 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.301350 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.301625 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.303557 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.303894 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.304618 3566 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.304713 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.305228 3566 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:39.305605 3566 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.305989 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.306694 1919 raft_consensus.cc:2366] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3.
I20250626 08:23:39.306843 3759 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.308773 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.243553 3821 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20250626 08:23:39.260324 3849 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.310444 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.260510 3886 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:39.309470 2586 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:39.321683 2582 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:39.334796 2585 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250626 08:23:39.419443 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 149819us
I20250626 08:23:39.321813 1967 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:39.340428 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.444453 2000 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
W20250626 08:23:39.483165 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 221565us
I20250626 08:23:39.482245 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.296581 3800 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.250174 3844 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:39.299044 3857 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.411648 3908 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.545040 1965 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:39.435801 3774 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:39.487593 3886 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.330648 3813 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.335587 3670 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.547894 3774 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.310056 3722 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.320225 3847 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.421020 3453 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
I20250626 08:23:39.561688 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.548946 3813 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:39.577867 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 237528us
I20250626 08:23:39.320767 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:39.550411 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 228682us
I20250626 08:23:39.579681 1846 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:39.550494 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.556304 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:39.334800 3939 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250626 08:23:39.408689 3566 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.584004 1847 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:39.554448 2551 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.415251 3759 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.255220 3715 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.592983 3566 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.578682 1968 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.594677 3715 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.595005 3844 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:39.595316 3851 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250626 08:23:39.595783 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 43814.3us
I20250626 08:23:39.595842 3851 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:39.596735 3851 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:39.582832 3861 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:39.598217 2580 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:39.598879 3861 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:39.599964 3861 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:39.600555 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 46228.8us
I20250626 08:23:39.601325 2551 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.562557 3738 raft_consensus.cc:2762] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250626 08:23:39.603543 3722 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:39.604269 3722 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:39.584076 2584 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:39.608641 3841 raft_consensus.cc:3053] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:39.609166 3841 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:39.587292 2582 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:39.608609 3851 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:39.612790 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.614248 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.618183 1890 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:39.588356 2585 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:39.589193 3848 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.590234 2586 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:39.622761 1841 raft_consensus.cc:2366] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5.
I20250626 08:23:39.588089 3934 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.594597 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:39.562992 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 289758us
I20250626 08:23:39.606153 2579 raft_consensus.cc:2391] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:23:39.642731 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 63101us
I20250626 08:23:39.592777 3817 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.416064 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.598151 3936 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.645457 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.662686 2587 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:39.645653 3918 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.647295 3886 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.664013 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.631531 3722 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20250626 08:23:39.672843 1965 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:39.650286 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 55798.4us
I20250626 08:23:39.651002 3848 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.653577 3774 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.656807 2550 leader_election.cc:400] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3.
I20250626 08:23:39.424423 3941 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20250626 08:23:39.471503 3920 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.624356 3851 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:39.666924 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:39.681102 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:39.681550 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:39.667587 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:39.681748 1842 raft_consensus.cc:2374] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4. Re-sending same reply.
I20250626 08:23:39.688473 2582 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:39.687214 1967 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:39.691421 1890 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:39.666213 3886 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.464167 1921 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:39.699112 1921 raft_consensus.cc:604] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:39.699589 1921 raft_consensus.cc:612] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:23:39.675458 2550 leader_election.cc:403] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250626 08:23:39.680122 1846 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:39.667208 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:39.680248 3774 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:39.703786 2586 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:39.672544 3791 raft_consensus.cc:695] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:39.707029 1967 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.674906 3714 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.680612 3918 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.707242 2002 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:39.698428 1968 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:39.714344 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 91562.3us
W20250626 08:23:39.720245 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 446328us
I20250626 08:23:39.722813 1967 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:39.704589 3934 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.724296 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:39.742197 2551 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:39.710345 3714 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:39.743901 1967 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.715020 3918 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.700428 3848 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250626 08:23:39.727217 3813 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3.
I20250626 08:23:39.750422 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:39.753620 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 210516us
I20250626 08:23:39.727157 3857 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.755277 1812 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:39.728406 3934 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:39.749965 3738 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:39.757700 3854 consensus_queue.cc:1046] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250626 08:23:39.758368 3738 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:39.758879 3738 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:39.759804 3791 consensus_queue.cc:237] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:39.745615 3851 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:39.762895 1816 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.748073 3714 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.761601 3851 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:39.770342 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.771167 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.771813 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.749212 3918 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:39.773377 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 88790.1us
I20250626 08:23:39.774664 1923 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:39.752604 3886 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:39.752722 3908 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:39.755326 3821 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.752969 3774 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.765043 3860 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:39.792268 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.785831 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:39.792622 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 45601.2us
I20250626 08:23:39.793094 1920 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:39.790189 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.794947 1927 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:39.764973 3857 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.797384 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.798266 1925 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:39.765931 3934 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.801652 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:39.801663 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.803910 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:39.804687 1919 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.755657 3847 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20250626 08:23:39.789121 3821 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:39.811774 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.796463 3844 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:39.796792 3666 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:39.792455 3860 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:39.823539 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:39.826397 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:39.801095 3817 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:39.831018 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 75815.2us
I20250626 08:23:39.818045 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:39.846940 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:39.847784 1995 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.796667 3906 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20250626 08:23:39.816653 3821 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.828114 3743 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:39.861851 1927 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:39.867904 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:39.868746 2001 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:39.840032 3666 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.842636 3860 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:39.843972 3851 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.876338 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:39.877429 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 65736.7us
I20250626 08:23:39.930128 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:39.932492 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.933053 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:39.810855 3727 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20250626 08:23:39.937081 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:39.863034 3886 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:39.935842 2000 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:39.939823 3952 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:39.878499 3666 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:39.874533 3920 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:39.874047 3844 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:39.931972 3784 raft_consensus.cc:3053] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
W20250626 08:23:39.944780 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 126897us
I20250626 08:23:39.954974 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:39.947818 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 183160us
I20250626 08:23:39.958613 1996 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:39.934271 3958 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:39.943506 3953 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:39.976653 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:39.987210 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:40.008854 2551 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:39.964224 3936 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:39.962798 3784 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:40.029489 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 60492us
W20250626 08:23:40.029366 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 285527us
W20250626 08:23:40.046411 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 106868us
I20250626 08:23:40.024349 3936 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.061028 1845 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
W20250626 08:23:40.035177 3849 consensus_meta.cc:287] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.502s user 0.007s sys 0.003s
I20250626 08:23:40.054250 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:39.967731 3952 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.067360 1845 raft_consensus.cc:604] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:40.027946 3957 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:39.975100 3958 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:40.081962 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 73090.1us
I20250626 08:23:40.083103 1814 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:40.082307 3920 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:40.085526 3920 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:40.084618 3963 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250626 08:23:40.087006 3920 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.087927 3813 raft_consensus.cc:3053] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:40.088310 3813 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:40.089140 3813 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.073370 3715 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:40.090291 3715 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.090688 3715 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:40.093691 3952 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:40.097478 3952 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:40.098040 3952 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:40.098415 3952 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:40.102265 3955 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20250626 08:23:40.061893 3886 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.108239 3841 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.109422 3841 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.109252 3952 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.066125 3849 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.111322 3920 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.112320 1814 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:40.064364 3936 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:40.116277 3958 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:40.109050 3738 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.122931 2001 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:40.066108 3908 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250626 08:23:40.136327 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 295939us
I20250626 08:23:40.132777 2587 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:40.133710 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:40.142093 3861 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:40.141969 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:40.142905 3861 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:40.143242 3851 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:40.144065 3861 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.150391 3965 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:40.153180 1967 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.154249 3965 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:40.154711 3965 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:40.155184 3965 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:40.141120 1812 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.139067 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 361) took 1026 ms. Trace:
I20250626 08:23:40.175737 3920 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:40.177119 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:40.188302 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 47234.8us
I20250626 08:23:40.190092 1812 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.191730 3968 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:40.192449 1812 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.197515 3906 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:40.199968 3906 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:40.206532 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:40.210500 3759 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:40.211666 3759 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:23:40.213500 3759 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:40.214241 3968 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:40.217754 3968 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:40.218837 3968 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.221208 3968 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.214860 3953 raft_consensus.cc:3053] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:40.222354 3953 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:40.223415 3953 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.225060 3953 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:40.226670 3759 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.227695 3759 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:40.229004 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.201893 3906 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250626 08:23:40.232457 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.239046 2585 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:40.235553 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.237815 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:40.239910 2587 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:40.243507 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.247020 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:40.251791 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 65043.8us
I20250626 08:23:40.252703 2587 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:40.138487 3743 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:40.253163 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 71303.6us
I20250626 08:23:40.171989 1928 rpcz_store.cc:276] 0626 08:23:39.112414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.112620 (+ 206us) service_pool.cc:225] Handling call
0626 08:23:39.121051 (+ 8431us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:39.125025 (+ 3974us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:23:39.160185 (+ 35160us) log.cc:844] Serialized 14 byte log entry
0626 08:23:39.758225 (+598040us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b18003f5920. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc306a 00007f4eb908960a
0626 08:23:40.134306 (+376081us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:40.134355 (+ 49us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:40.134418 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:40.134499 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:40.138994 (+ 4495us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":30501,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":88720,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7022,"spinlock_wait_cycles":29778816,"thread_start_us":375790,"threads_started":2,"wal-append.queue_time_us":961471}
W20250626 08:23:40.256036 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 74592.4us
I20250626 08:23:40.138223 3908 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.262092 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:40.248536 3813 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.271790 1893 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.118726 3784 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.119745 3844 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.254851 3920 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:40.248629 3971 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:40.257229 3970 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:40.169637 3965 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:40.282891 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 318105us
I20250626 08:23:40.286978 1816 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:40.263752 3908 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.300191 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.300863 1996 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 1.
I20250626 08:23:40.305368 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.307446 1893 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.314121 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.314373 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.314226 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.316226 1999 raft_consensus.cc:2366] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
W20250626 08:23:40.316826 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 176820us
I20250626 08:23:40.295902 3970 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.322189 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.319407 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:40.321641 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.326200 1999 raft_consensus.cc:2366] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:40.326017 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.327106 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.327900 1999 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:40.269268 3906 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20250626 08:23:40.330816 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.333278 1848 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250626 08:23:40.340137 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.337242 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.313079 3851 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:40.315428 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.348169 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.349956 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.351493 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.351778 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.352511 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.353353 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.354056 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:40.356060 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:40.356309 3923 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.323s user 0.002s sys 0.002s
I20250626 08:23:40.356241 3666 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:40.357246 3966 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20250626 08:23:40.357064 3743 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.318091 3971 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250626 08:23:40.358271 3961 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250626 08:23:40.358173 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.426770 1846 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:40.359323 3738 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:40.299834 3920 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.367810 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.428781 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:40.336812 3978 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:40.335268 3970 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:40.363430 1919 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.555s user 0.005s sys 0.003s
I20250626 08:23:40.440948 1919 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:40.376020 1842 raft_consensus.cc:2366] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
W20250626 08:23:40.376497 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:40.334561 3981 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:40.378677 3908 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:40.423025 3934 consensus_meta.cc:287] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.614s user 0.005s sys 0.005s
I20250626 08:23:40.380107 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.380751 1843 raft_consensus.cc:2366] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5.
I20250626 08:23:40.381201 1848 raft_consensus.cc:2366] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4.
I20250626 08:23:40.363798 3988 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:40.385825 3851 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.390199 3983 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election won for term 2
W20250626 08:23:40.397007 3918 consensus_meta.cc:287] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.596s user 0.007s sys 0.000s
I20250626 08:23:40.402467 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:40.403244 3959 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
W20250626 08:23:40.404053 3960 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.329s user 0.000s sys 0.001s
I20250626 08:23:40.404796 3821 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:40.407091 3857 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.598s user 0.007s sys 0.002s
I20250626 08:23:40.408648 3965 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:40.408996 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 83133.9us
I20250626 08:23:40.330827 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 08:23:40.513746 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:40.416446 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:40.422314 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 228794us
W20250626 08:23:40.423827 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:40.422537 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 113457us
I20250626 08:23:40.515936 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.517177 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:40.424590 3958 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:40.526860 3727 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:40.527334 3727 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:40.527455 3958 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:40.528164 3958 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:40.425192 3849 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:40.473943 1999 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:40.531939 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.532694 1999 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:40.476356 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:40.534202 2582 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250626 08:23:40.534552 3989 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
W20250626 08:23:40.534982 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 273) took 1490 ms (client timeout 1550 ms). Trace:
W20250626 08:23:40.535526 2002 rpcz_store.cc:269] 0626 08:23:39.044098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.177708 (+133610us) service_pool.cc:225] Handling call
0626 08:23:40.534766 (+1357058us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b540018ea54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:40.534908 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1066099072}
I20250626 08:23:40.529209 1968 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:40.537045 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.422521 3971 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:40.477732 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.478091 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:40.477782 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:40.441107 3848 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.034s sys 0.019s
I20250626 08:23:40.482749 1927 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:40.397078 3715 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.486323 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.488230 2586 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:40.493925 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.495033 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.500836 3821 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:40.500945 3908 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:40.503338 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.504194 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:40.442540 3861 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:40.425391 3860 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.501705 3857 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.518433 1926 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250626 08:23:40.518958 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:40.519019 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:40.520005 3973 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:40.521081 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:40.485666 3791 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.008s sys 0.005s
I20250626 08:23:40.429661 3952 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:40.548233 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 217718us
I20250626 08:23:40.534302 1918 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.533740 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:40.593199 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:40.597424 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 67047.6us
W20250626 08:23:40.599438 1995 consensus_meta.cc:287] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.751s user 0.014s sys 0.009s
I20250626 08:23:40.440793 3759 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.496171 3983 raft_consensus.cc:695] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:40.476958 3934 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:40.484700 3918 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:40.491052 3988 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:40.599867 1850 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:40.637753 1998 raft_consensus.cc:2366] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:41.022912 1999 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:40.744093 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.024196 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.024920 1999 raft_consensus.cc:2374] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3. Re-sending same reply.
I20250626 08:23:41.025674 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:40.710042 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.026185 1999 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.662452 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.028872 1848 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:41.031216 1919 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:40.669637 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 275234us
I20250626 08:23:40.615046 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:41.027513 1925 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.001s sys 0.008s
W20250626 08:23:41.037657 1925 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.001s sys 0.009s
W20250626 08:23:40.542327 3714 consensus_meta.cc:287] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.751s user 0.010s sys 0.001s
W20250626 08:23:40.664693 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 135542us
I20250626 08:23:41.038697 1995 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:23:40.751991 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 394934us
I20250626 08:23:40.936962 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:41.023629 1924 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:23:41.040303 3861 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:41.046038 3813 consensus_meta.cc:287] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.768s user 0.010s sys 0.000s
W20250626 08:23:41.046339 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 568) took 1973 ms (client timeout 1620 ms). Trace:
W20250626 08:23:41.046794 2579 rpcz_store.cc:269] 0626 08:23:39.061317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.063505 (+ 2188us) service_pool.cc:225] Handling call
0626 08:23:41.009927 (+1946422us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b540050da54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:41.034941 (+ 25014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1119578880}
I20250626 08:23:41.047394 3861 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.047940 3861 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:41.048363 3936 consensus_meta.cc:287] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.863s user 0.011s sys 0.000s
I20250626 08:23:41.048846 3936 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.049616 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1773) took 1267 ms. Trace:
W20250626 08:23:41.026593 3953 consensus_meta.cc:287] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.801s user 0.003s sys 0.013s
I20250626 08:23:41.050021 1925 rpcz_store.cc:276] 0626 08:23:39.782208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.799140 (+ 16932us) service_pool.cc:225] Handling call
0626 08:23:39.809773 (+ 10633us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:39.810067 (+ 294us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:40.171784 (+361717us) log.cc:844] Serialized 14 byte log entry
0626 08:23:41.038352 (+866568us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:41.038406 (+ 54us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:41.049372 (+ 10966us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.049451 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:41.049564 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55510,"prepare.queue_time_us":132536,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":868,"raft.queue_time_us":71512,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":20777,"spinlock_wait_cycles":22115072,"thread_start_us":1106222,"threads_started":3,"wal-append.queue_time_us":338523}
I20250626 08:23:41.048444 3861 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.054240 1812 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:41.055023 3715 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:40.950717 3970 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.056156 1849 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:41.057145 3970 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:41.062731 3997 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20250626 08:23:40.614203 3738 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.449s
W20250626 08:23:40.598294 3848 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.034s sys 0.019s
I20250626 08:23:41.066802 3668 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:41.067140 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:41.039126 3844 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.000s sys 0.016s
W20250626 08:23:41.069208 3844 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.000s sys 0.016s
I20250626 08:23:41.011474 3920 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:41.090404 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 569338us
I20250626 08:23:41.091049 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:41.092407 1845 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:41.096318 1918 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:41.100502 1848 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:41.102003 2000 raft_consensus.cc:2391] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:40.948843 3996 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20250626 08:23:41.026829 3791 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.070s user 0.023s sys 0.016s
W20250626 08:23:40.675451 3972 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.425s user 0.000s sys 0.001s
I20250626 08:23:41.041648 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:40.598943 3981 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:41.119521 1841 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:41.120127 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.154675 2001 raft_consensus.cc:2366] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:41.043983 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 281) took 1214 ms. Trace:
I20250626 08:23:41.178378 1995 rpcz_store.cc:276] 0626 08:23:39.829204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.838565 (+ 9361us) service_pool.cc:225] Handling call
0626 08:23:41.043936 (+1205371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:41.133112 1846 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:40.982870 3965 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:40.908973 3666 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:41.184064 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:41.172667 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 131112us
I20250626 08:23:40.527782 3727 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:40.895197 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:41.169548 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:41.203429 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.335976 1998 raft_consensus.cc:2374] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4. Re-sending same reply.
I20250626 08:23:41.240645 1850 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:41.337940 1927 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:41.338608 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:41.328279 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:41.341207 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 287037us
W20250626 08:23:41.341516 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 298165us
W20250626 08:23:41.259896 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.009s sys 0.009s
W20250626 08:23:41.343812 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.804s user 0.009s sys 0.010s
I20250626 08:23:41.344061 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:41.345172 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:41.334695 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 221448us
I20250626 08:23:40.596463 3971 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:40.945534 3994 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:41.213579 1999 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:41.262451 2002 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
W20250626 08:23:41.274134 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 229346us
I20250626 08:23:41.342104 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:41.123615 3918 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.177435 3981 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:41.349606 1968 leader_election.cc:400] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5.
I20250626 08:23:41.358139 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.355850 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.364178 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:41.176314 3958 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.648s user 0.007s sys 0.004s
I20250626 08:23:41.346909 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:41.119664 3759 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.131551 3988 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.117673 3936 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.122088 3934 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:41.189424 3841 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.080s user 0.009s sys 0.004s
I20250626 08:23:41.424726 3993 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:41.424952 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.042693 3973 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.430080 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:41.510664 1920 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:41.431926 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 88371.2us
I20250626 08:23:41.130914 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 578) took 1055 ms. Trace:
W20250626 08:23:41.433643 1997 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.012s sys 0.026s
I20250626 08:23:41.434531 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1631) took 1038 ms. Trace:
W20250626 08:23:41.367729 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:41.553591 1997 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.939s user 0.013s sys 0.027s
I20250626 08:23:41.553545 1968 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:41.106171 3714 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.347463 3998 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:41.435803 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.137048 3978 consensus_queue.cc:1046] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.292s
I20250626 08:23:41.175557 1994 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:41.112476 3866 raft_consensus.cc:2762] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20250626 08:23:41.489647 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1606) took 1228 ms. Trace:
I20250626 08:23:41.490481 4018 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:41.583971 2001 rpcz_store.cc:276] 0626 08:23:40.261644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.261788 (+ 144us) service_pool.cc:225] Handling call
0626 08:23:41.489423 (+1227635us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b540018d654. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:41.489581 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2048785664}
I20250626 08:23:41.584733 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:41.449975 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.464339 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:41.436622 1927 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:41.586215 1999 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:41.586572 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1710) took 1045 ms. Trace:
I20250626 08:23:41.586959 1927 rpcz_store.cc:276] 0626 08:23:40.541220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.102149 (+560929us) service_pool.cc:225] Handling call
0626 08:23:41.586507 (+484358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:41.510295 3774 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.328897 2579 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:41.509191 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.122323 4008 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:41.180702 4001 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:41.175374 3821 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.554900 1995 rpcz_store.cc:276] 0626 08:23:40.396422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.363812 (+967390us) service_pool.cc:225] Handling call
0626 08:23:41.434452 (+ 70640us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:41.360908 4019 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:41.556699 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 209949us
I20250626 08:23:41.364903 3970 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:41.429524 3920 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250626 08:23:41.466614 3968 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.244s user 0.011s sys 0.003s
I20250626 08:23:41.556015 1814 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:41.574760 3998 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.435287 3981 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:41.569482 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:41.348349 3861 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.502312 3993 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:41.510421 3715 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.588048 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.624898 1927 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:41.449255 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.626240 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:41.589382 1967 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:41.605619 3714 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.546236 3847 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.437428 3738 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.595360 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.633584 1999 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:41.634353 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1936) took 1049 ms. Trace:
I20250626 08:23:41.560107 3841 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:41.641671 1918 raft_consensus.cc:1238] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:41.443014 3958 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.591465 2002 raft_consensus.cc:2374] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4. Re-sending same reply.
I20250626 08:23:41.602473 1968 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:41.597318 3774 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.604527 4024 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:41.604393 3851 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:41.605309 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.820609 2001 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:41.820999 3774 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:41.821110 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1935) took 1237 ms. Trace:
I20250626 08:23:41.821589 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 579) took 1700 ms. Trace:
I20250626 08:23:41.821549 2001 rpcz_store.cc:276] 0626 08:23:40.584181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.593769 (+1009588us) service_pool.cc:225] Handling call
0626 08:23:41.821047 (+227278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:41.645562 3738 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.609110 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.612032 1923 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:41.613231 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 58745.2us
I20250626 08:23:41.588688 4003 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.381494 1993 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:41.598884 3936 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:41.589800 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 247773us
W20250626 08:23:41.834244 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 231844us
I20250626 08:23:41.834431 1812 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:41.835134 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:41.635658 1845 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1603) took 1330 ms (client timeout 1743 ms). Trace:
I20250626 08:23:41.636723 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:41.361399 3965 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.634760 1999 rpcz_store.cc:276] 0626 08:23:40.584481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.595080 (+1010599us) service_pool.cc:225] Handling call
0626 08:23:41.634272 (+ 39192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:41.693727 4001 consensus_queue.cc:1046] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.132s
I20250626 08:23:41.838485 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.436968 3983 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.647969 4014 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.654742 1918 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250626 08:23:41.658492 3722 raft_consensus.cc:2762] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250626 08:23:41.659543 3947 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 1.932s user 0.007s sys 0.004s
I20250626 08:23:41.817162 4008 raft_consensus.cc:695] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:41.662381 1927 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:41.629458 3993 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:41.596911 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:41.624924 3968 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:41.874521 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:41.601564 3821 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.697194 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:41.637965 3861 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:41.594106 4018 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.631659 3908 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20250626 08:23:41.631525 3715 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:41.732204 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:41.905721 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:41.698134 3860 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.822644 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:41.843662 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1939) took 1224 ms (client timeout 1538 ms). Trace:
I20250626 08:23:41.844427 3988 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250626 08:23:41.605489 3967 log.cc:927] Time spent T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.521s user 0.000s sys 0.020s
I20250626 08:23:41.602058 4019 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.642975 3847 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.826145 1995 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:41.822041 2586 rpcz_store.cc:276] 0626 08:23:40.121067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.121195 (+ 128us) service_pool.cc:225] Handling call
0626 08:23:40.135275 (+ 14080us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:40.599134 (+463859us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.821365 (+1222231us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b540013bd54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:41.821528 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":811112704}
W20250626 08:23:41.817668 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 250091us
I20250626 08:23:41.713205 3998 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.921973 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.922736 2001 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:41.830580 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:41.923619 1923 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:41.599203 3906 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:41.835726 1845 rpcz_store.cc:269] 0626 08:23:40.304974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.305123 (+ 149us) service_pool.cc:225] Handling call
0626 08:23:41.635447 (+1330324us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54002f30d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:41.635589 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1711860480}
W20250626 08:23:41.836467 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 237603us
W20250626 08:23:41.842343 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:41.823304 3920 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:41.823715 4024 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:41.720429 3841 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.659891 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1708) took 1161 ms. Trace:
W20250626 08:23:41.867298 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:41.730090 3985 consensus_queue.cc:1046] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.175s
W20250626 08:23:41.870771 4020 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.306s user 0.001s sys 0.000s
I20250626 08:23:41.842335 4017 consensus_queue.cc:1046] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20250626 08:23:41.938015 3906 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.938266 3953 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.938632 3906 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.938910 3953 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.912216 3861 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:41.842202 3952 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:41.946568 3952 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:41.921237 2579 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:41.946995 3715 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.877653 3848 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20250626 08:23:41.947814 3666 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.947525 3952 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.948300 3666 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:41.876425 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 64505.9us
I20250626 08:23:41.950263 3666 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.875663 1814 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:41.882421 2002 raft_consensus.cc:2391] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:41.883013 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:41.884071 3973 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:41.884497 4009 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250626 08:23:41.885020 3821 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.885943 1741 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32876 (request call id 296) took 7390 ms. Trace:
I20250626 08:23:41.834765 3714 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:41.886139 3965 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.873355 3971 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.898559 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:41.907456 1967 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:41.970012 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 62681.7us
W20250626 08:23:41.970316 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 71825.8us
I20250626 08:23:41.970829 1967 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:41.840659 3851 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:41.916760 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:41.977757 1842 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:41.978093 2582 raft_consensus.cc:2391] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:41.887418 3936 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:41.875377 3981 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.934538 4030 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:41.926789 1924 rpcz_store.cc:276] 0626 08:23:40.498373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.044835 (+546462us) service_pool.cc:225] Handling call
0626 08:23:41.089033 (+ 44198us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:23:41.360249 (+271216us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:41.360293 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:23:41.363411 (+ 3118us) log.cc:844] Serialized 52 byte log entry
0626 08:23:41.590745 (+227334us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:41.590861 (+ 116us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.590992 (+ 131us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:41.657735 (+ 66743us) raft_consensus.cc:1683] finished
0626 08:23:41.657757 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:41.659819 (+ 2062us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":202798,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":163,"thread_start_us":364006,"threads_started":2,"wal-append.queue_time_us":222643}
W20250626 08:23:41.935561 4026 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250626 08:23:41.936431 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1618) took 1598 ms. Trace:
W20250626 08:23:41.937227 3727 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.001s
I20250626 08:23:41.553385 2578 rpcz_store.cc:276] 0626 08:23:40.076010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.078463 (+ 2453us) service_pool.cc:225] Handling call
0626 08:23:40.094300 (+ 15837us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:40.137838 (+ 43538us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:40.139337 (+ 1499us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:40.286738 (+147401us) log.cc:844] Serialized 26 byte log entry
0626 08:23:40.981599 (+694861us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:41.034805 (+ 53206us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.059004 (+ 24199us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:41.130625 (+ 71621us) raft_consensus.cc:1683] finished
0626 08:23:41.130656 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:41.130839 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25797,"raft.queue_time_us":36499,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":4467,"thread_start_us":310736,"threads_started":1,"wal-append.queue_time_us":805291}
I20250626 08:23:41.940892 1893 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:41.941020 3906 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:41.942123 4029 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:41.941550 3861 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.942445 4008 consensus_queue.cc:237] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.943338 3958 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:41.943825 3738 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:41.902521 3994 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:41.945266 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 379) took 2191 ms. Trace:
W20250626 08:23:42.034288 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 93474.5us
I20250626 08:23:41.951682 2585 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:41.951845 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:41.905000 3968 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:41.951200 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 75641.8us
I20250626 08:23:41.964280 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:42.036623 1995 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:42.037003 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1937) took 1452 ms. Trace:
I20250626 08:23:42.037278 1995 rpcz_store.cc:276] 0626 08:23:40.584771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.933662 (+1348891us) service_pool.cc:225] Handling call
0626 08:23:42.036948 (+103286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:42.030303 4029 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:42.037966 4038 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:42.038595 4038 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.038872 4029 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.039222 3968 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.042155 3953 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:41.967626 3886 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.861s user 0.015s sys 0.017s
I20250626 08:23:41.976318 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:41.934015 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:41.985416 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:42.046670 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:41.929538 4024 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:41.963445 3970 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:42.019456 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1619) took 1627 ms. Trace:
I20250626 08:23:42.057065 1997 rpcz_store.cc:276] 0626 08:23:40.392621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.411986 (+ 19365us) service_pool.cc:225] Handling call
0626 08:23:40.433160 (+ 21174us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:40.470958 (+ 37798us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:40.471008 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:40.493067 (+ 22059us) log.cc:844] Serialized 26 byte log entry
0626 08:23:41.554208 (+1061141us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:41.921386 (+367178us) log.cc:844] Serialized 14 byte log entry
0626 08:23:41.930455 (+ 9069us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.930556 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:42.019196 (+ 88640us) raft_consensus.cc:1683] finished
0626 08:23:42.019219 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:42.019389 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87539,"prepare.queue_time_us":347872,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":38980,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":44429,"spinlock_wait_cycles":1638144,"thread_start_us":1283886,"threads_started":2,"wal-append.queue_time_us":1089927}
I20250626 08:23:41.994616 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:42.020836 2551 leader_election.cc:336] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.847s (SENT)
I20250626 08:23:41.876875 4014 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.994128 3936 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:41.991746 1996 rpcz_store.cc:276] 0626 08:23:40.338396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.377409 (+ 39013us) service_pool.cc:225] Handling call
0626 08:23:40.378012 (+ 603us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:40.849471 (+471459us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:40.849521 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:41.043448 (+193927us) log.cc:844] Serialized 26 byte log entry
0626 08:23:41.365332 (+321884us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:41.615482 (+250150us) log.cc:844] Serialized 14 byte log entry
0626 08:23:41.616411 (+ 929us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.616483 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:41.936201 (+319718us) raft_consensus.cc:1683] finished
0626 08:23:41.936221 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:41.936369 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":466562,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":68919,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":31004,"thread_start_us":576886,"threads_started":3,"wal-append.queue_time_us":131971}
I20250626 08:23:41.991415 3988 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20250626 08:23:41.929785 3956 consensus_queue.cc:1046] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20250626 08:23:41.960865 1741 rpcz_store.cc:276] 0626 08:23:34.495735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.496735 (+ 1000us) service_pool.cc:225] Handling call
0626 08:23:36.474248 (+1977513us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b54001df954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003a70 00007f4eb9087b1a 00007f4ebed9afac 00007f4ebee937ee 00007f4ebd3dd624 00007f4ebd3dd5b2 00007f4ebd3dd4e1 00007f4ebd3dd45c 00007f4ebd3dc712 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:41.885860 (+5411612us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:39.128705 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:39.337753 (+209048us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:39.337806 (+ 53us) write_op.cc:435] Acquired schema lock
0626 08:23:39.337838 (+ 32us) tablet.cc:660] Decoding operations
0626 08:23:39.338311 (+ 473us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:39.338381 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:39.338410 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0626 08:23:39.338704 (+ 294us) tablet.cc:699] Row locks acquired
0626 08:23:39.338736 (+ 32us) write_op.cc:260] PREPARE: finished
0626 08:23:39.338910 (+ 174us) write_op.cc:270] Start()
0626 08:23:39.339060 (+ 150us) write_op.cc:276] Timestamp: P: 1750926219338833 usec, L: 0
0626 08:23:39.339080 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 08:23:39.339858 (+ 778us) log.cc:844] Serialized 541 byte log entry
0626 08:23:40.499226 (+1159368us) op_driver.cc:464] REPLICATION: finished
0626 08:23:41.110799 (+611573us) write_op.cc:301] APPLY: starting
0626 08:23:41.110931 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:41.111094 (+ 163us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:41.111290 (+ 196us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:41.111715 (+ 425us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:41.112055 (+ 340us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0626 08:23:41.112147 (+ 92us) write_op.cc:312] APPLY: finished
0626 08:23:41.112446 (+ 299us) log.cc:844] Serialized 23 byte log entry
0626 08:23:41.112687 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:41.112771 (+ 84us) write_op.cc:454] Released schema lock
0626 08:23:41.112820 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0626 08:23:41.113163 (+ 343us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:41.113191 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":278344,"reported_tablets":2,"spinlock_wait_cycles":496040704,"child_traces":[["op",{"apply.queue_time_us":611151,"apply.run_cpu_time_us":2401,"apply.run_wall_time_us":2402,"num_ops":1,"prepare.queue_time_us":170060,"prepare.run_cpu_time_us":18315,"prepare.run_wall_time_us":559580,"raft.queue_time_us":651478,"raft.run_cpu_time_us":16250,"raft.run_wall_time_us":598334,"replication_time_us":1135974,"spinlock_wait_cycles":23018752,"thread_start_us":1022043,"threads_started":4,"wal-append.queue_time_us":428355,"wal-append.run_wall_time_us":1265251}]]}
I20250626 08:23:42.023545 1968 leader_election.cc:400] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:42.023893 3981 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:41.978783 3971 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:41.835716 4003 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:41.917523 1993 rpcz_store.cc:269] 0626 08:23:40.614841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.698112 (+ 83271us) service_pool.cc:225] Handling call
0626 08:23:41.839103 (+1140991us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:42.026633 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.028689 1928 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:41.933259 3920 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.991279 3965 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:41.984413 4001 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:41.940510 3985 raft_consensus.cc:3053] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:42.035308 1848 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250626 08:23:42.047402 2586 raft_consensus.cc:2408] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250626 08:23:42.052085 1816 leader_election.cc:336] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.536s (SENT)
I20250626 08:23:42.054720 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:41.977164 3714 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.056031 2001 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:42.013289 3727 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:41.994189 3934 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:42.004782 4030 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.111860 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.303244 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.040258 4029 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:42.257774 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9808848200000284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20250626 08:23:42.364459 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9457848199999717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
W20250626 08:23:42.365067 1893 connection.cc:232] RPC call timeout handler was delayed by -1.93858481999996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
I20250626 08:23:42.263252 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.034037 1921 rpcz_store.cc:276] 0626 08:23:39.754104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.754293 (+ 189us) service_pool.cc:225] Handling call
0626 08:23:39.754624 (+ 331us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:39.758688 (+ 4064us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:39.758726 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:39.759366 (+ 640us) log.cc:844] Serialized 26 byte log entry
0626 08:23:39.959440 (+200074us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:39.982049 (+ 22609us) log.cc:844] Serialized 14 byte log entry
0626 08:23:39.982350 (+ 301us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:39.982450 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:40.103688 (+121238us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b1800468ac0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:23:41.512311 (+1408623us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:41.893200 (+380889us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b1800468ac0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:23:41.945044 (+ 51844us) raft_consensus.cc:1683] finished
0626 08:23:41.945079 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:41.945206 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":92789,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":150188,"raft.run_cpu_time_us":523,"raft.run_wall_time_us":34359,"spinlock_wait_cycles":159538048,"thread_start_us":198564,"threads_started":1,"wal-append.queue_time_us":287981}
I20250626 08:23:42.303329 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:42.175992 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.349898 1968 leader_election.cc:403] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250626 08:23:42.320850 1920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1783) took 1874 ms (client timeout 2080 ms). Trace:
I20250626 08:23:42.042475 4027 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:42.370431 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:42.370890 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 347479us
I20250626 08:23:42.371896 1968 leader_election.cc:400] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5.
I20250626 08:23:42.177049 1997 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
W20250626 08:23:42.071095 2000 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 288) took 2106 ms (client timeout 2245 ms). Trace:
I20250626 08:23:42.372834 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1621) took 1979 ms. Trace:
I20250626 08:23:42.373411 1997 rpcz_store.cc:276] 0626 08:23:40.393389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.069750 (+1676361us) service_pool.cc:225] Handling call
0626 08:23:42.176870 (+107120us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.372609 (+195739us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.372782 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:42.352283 1923 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:42.352816 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1943) took 1306 ms. Trace:
I20250626 08:23:42.352836 1845 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:42.191270 1812 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.354019 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:42.375545 2001 rpcz_store.cc:276] 0626 08:23:41.046069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.921682 (+875613us) service_pool.cc:225] Handling call
0626 08:23:42.352747 (+431065us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:42.376266 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 185128us
I20250626 08:23:42.376543 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:42.352404 1848 pending_rounds.cc:85] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting all ops after (but not including) 4
I20250626 08:23:42.376142 2550 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:42.365331 1893 connection.cc:232] RPC call timeout handler was delayed by -1.723484820000067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20250626 08:23:42.029554 4018 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:42.341321 1814 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 2.080s (SENT) [suppressed 31 similar messages]
W20250626 08:23:42.366654 1920 rpcz_store.cc:269] 0626 08:23:40.446449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.446625 (+ 176us) service_pool.cc:225] Handling call
0626 08:23:42.320669 (+1874044us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54002e04d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:42.320789 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2115032448}
I20250626 08:23:42.371860 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:42.304140 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1781) took 1858 ms. Trace:
I20250626 08:23:42.373021 1968 leader_election.cc:400] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4.
I20250626 08:23:42.257251 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.351766 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:42.373106 2000 rpcz_store.cc:269] 0626 08:23:39.943142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.943278 (+ 136us) service_pool.cc:225] Handling call
0626 08:23:42.049019 (+2105741us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400131fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:42.049199 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2624359680}
W20250626 08:23:42.303972 3964 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.186s user 0.000s sys 0.021s
I20250626 08:23:42.404583 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.405094 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1623) took 2011 ms. Trace:
I20250626 08:23:42.404943 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.377575 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.084774 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.407740 1999 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:42.060441 3886 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.940s user 0.015s sys 0.018s
I20250626 08:23:42.363822 3985 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:42.406250 1926 rpcz_store.cc:276] 0626 08:23:40.445707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.445847 (+ 140us) service_pool.cc:225] Handling call
0626 08:23:40.446327 (+ 480us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:41.022662 (+576335us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.303877 (+1281215us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b54002af3d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:42.304077 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":129085184}
I20250626 08:23:42.383702 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:42.378340 1893 connection.cc:232] RPC call timeout handler was delayed by -1.6622189149998121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20250626 08:23:42.415552 1893 connection.cc:232] RPC call timeout handler was delayed by -1.6577189149998048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20250626 08:23:42.416043 1893 connection.cc:232] RPC call timeout handler was delayed by -1.6064189149997219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20250626 08:23:42.416415 1893 connection.cc:232] RPC call timeout handler was delayed by -1.604618914999719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
W20250626 08:23:42.416795 1893 connection.cc:232] RPC call timeout handler was delayed by -1.5992189149997103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20250626 08:23:42.417160 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3886189149998245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20250626 08:23:42.407919 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:42.379209 1848 pending_rounds.cc:107] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250626 08:23:42.417943 1848 raft_consensus.cc:2887] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:42.351673 3965 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:42.430847 3965 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:42.436228 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 384174us
W20250626 08:23:42.383961 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 363130us
I20250626 08:23:42.345681 4001 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:42.408916 1968 leader_election.cc:403] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250626 08:23:42.380110 2002 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:42.410109 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.441449 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.366099 4038 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.157004 3970 raft_consensus.cc:695] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:42.410818 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.320266 3841 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.319947 4019 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:42.042194 3994 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.405539 2001 rpcz_store.cc:276] 0626 08:23:40.393992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.404285 (+2010293us) service_pool.cc:225] Handling call
0626 08:23:42.404609 (+ 324us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.404802 (+ 193us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.404826 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:42.404865 (+ 39us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:42.404893 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.404945 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:42.405045 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:42.406627 1850 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:42.377251 4027 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:42.438416 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:42.442559 1845 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:42.403824 3942 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:42.379444 3971 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250626 08:23:42.445180 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 72284.3us
I20250626 08:23:42.447345 1923 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:42.409103 3908 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:42.379818 3861 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:42.370250 3934 raft_consensus.cc:695] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:42.452030 1812 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.453476 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:42.426606 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3733189149997997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
W20250626 08:23:42.455111 1816 leader_election.cc:336] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.538s (SENT)
W20250626 08:23:42.551255 1893 connection.cc:232] RPC call timeout handler was delayed by -1.2374189150000348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
I20250626 08:23:42.551510 1816 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:42.472188 1924 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:42.552469 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.555539 1924 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:23:42.502920 3997 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20250626 08:23:42.512041 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:42.516755 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.563441 1843 raft_consensus.cc:2366] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:42.519888 1968 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:42.564245 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 112343us
I20250626 08:23:42.522804 1920 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:42.439370 2578 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:42.533428 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:42.538365 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:42.549332 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 375) took 2868 ms. Trace:
I20250626 08:23:42.446564 4001 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:42.472852 1926 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:42.574185 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.561813 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:42.564612 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 44885.3us
I20250626 08:23:42.453948 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:42.564630 1893 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.574026 3973 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:42.582423 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.582576 3973 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:42.583580 3973 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:42.584582 1893 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:42.586431 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.448565 3886 consensus_queue.cc:1046] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20250626 08:23:42.587811 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.588099 3759 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:42.588315 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 141934us
I20250626 08:23:42.588482 1843 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:23:42.589187 1816 leader_election.cc:336] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.589s (SENT)
I20250626 08:23:42.589740 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:42.590729 3957 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:42.591187 3957 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:42.591543 3957 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:42.591806 1816 leader_election.cc:336] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.640s (SENT)
I20250626 08:23:42.559394 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.572166 1845 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:42.494818 3988 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:42.455281 3738 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4.
W20250626 08:23:42.572422 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 127877us
I20250626 08:23:42.497284 4030 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:42.570266 1847 rpcz_store.cc:276] 0626 08:23:39.654009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:39.655993 (+ 1984us) service_pool.cc:225] Handling call
0626 08:23:39.656424 (+ 431us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:39.684609 (+ 28185us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:39.684659 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:39.705746 (+ 21087us) log.cc:844] Serialized 26 byte log entry
0626 08:23:39.941811 (+236065us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:39.941848 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:39.941934 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:42.303250 (+2361316us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:42.495720 (+192470us) raft_consensus.cc:1683] finished
0626 08:23:42.495753 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:42.522282 (+ 26529us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6564,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":16324,"spinlock_wait_cycles":45874432,"thread_start_us":198169,"threads_started":1,"wal-append.queue_time_us":351156}
I20250626 08:23:42.525044 3942 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:42.575400 1848 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:42.598237 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 256958us
I20250626 08:23:42.596374 1893 leader_election.cc:400] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:42.595157 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.516144 4027 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:42.589593 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.616763 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:42.617714 3738 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:42.618458 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.568559 3860 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:42.573742 3668 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:42.623570 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.626484 1848 raft_consensus.cc:3053] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:42.626850 1848 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:42.622169 3714 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.598953 3813 consensus_queue.cc:1046] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20250626 08:23:42.620330 1850 raft_consensus.cc:2391] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:42.626046 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 61173.4us
I20250626 08:23:42.593027 4038 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:42.577265 3817 consensus_queue.cc:1046] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20250626 08:23:42.601370 3942 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:42.659665 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:42.623458 2585 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.581s user 0.000s sys 0.014s
W20250626 08:23:42.652438 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 56179.3us
I20250626 08:23:42.663172 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:23:42.601851 3952 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.037s sys 0.097s
W20250626 08:23:42.629017 4037 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.413s user 0.000s sys 0.010s
I20250626 08:23:42.657294 1848 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.668856 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.687592 2585 raft_consensus.cc:2408] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:23:42.702248 1847 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:42.690910 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:42.707729 2582 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:42.708062 1893 leader_election.cc:400] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:42.702492 1848 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:42.662113 3993 consensus_meta.cc:287] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.784s user 0.012s sys 0.000s
I20250626 08:23:42.614538 3988 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.712105 1890 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:42.690250 3952 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.038s sys 0.102s
I20250626 08:23:42.713562 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:42.717901 1893 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.620263 3970 consensus_queue.cc:237] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.618683 4030 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
W20250626 08:23:42.692224 4044 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20250626 08:23:42.571262 3985 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.722982 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.634042 3738 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.728753 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.692567 3936 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:42.710568 3908 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:42.690049 3727 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.746107 1848 raft_consensus.cc:1273] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250626 08:23:42.713538 4018 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:42.729650 4031 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:42.752920 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:42.725809 4041 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.467s user 0.006s sys 0.005s
I20250626 08:23:42.752522 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.754127 4008 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:42.754751 4008 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:42.755249 4008 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:42.755399 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.757050 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.758440 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.759074 1843 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:42.759742 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:42.761844 1893 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:42.771370 1843 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:23:42.775177 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:42.775020 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.780516 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.780736 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.781733 2584 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:42.769714 4008 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.782066 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:42.718675 4003 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:42.724129 3860 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:42.784942 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 168256us
I20250626 08:23:42.724458 4019 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:42.787295 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.724707 3920 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.788849 2583 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:42.736635 3841 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:42.714815 3993 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.746834 4017 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:42.812901 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.748582 1993 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:42.788872 3985 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:42.753479 4031 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:42.798144 3908 raft_consensus.cc:3053] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:42.795848 4019 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:42.801985 4043 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.353s user 0.000s sys 0.001s
I20250626 08:23:42.796005 3817 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:42.805552 4053 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:42.825695 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:42.831961 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 181250us
W20250626 08:23:42.835484 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 59822.6us
W20250626 08:23:42.838723 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 76913.4us
I20250626 08:23:42.800966 3727 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:42.844911 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.867136 1850 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:42.807054 3920 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:42.804462 4014 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:42.799530 3860 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.799196 4003 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:42.864933 1893 leader_election.cc:400] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:42.727329 3934 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:42.871069 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.841960 4031 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:42.873363 1893 leader_election.cc:403] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:23:42.843191 3908 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:42.847167 4019 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:42.866456 3727 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:42.850440 3817 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:42.880089 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:42.903976 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:42.905543 1845 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:42.880120 1965 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.881287 2551 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:42.883108 1893 leader_election.cc:400] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:42.883925 3908 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.886868 4018 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:23:42.887140 3993 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:42.870733 4008 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:42.903888 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.880410 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.875636 4014 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:42.913630 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 79398us
I20250626 08:23:42.965277 1968 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:42.969105 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 382) took 2589 ms. Trace:
I20250626 08:23:42.969534 4064 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:23:42.969749 1841 rpcz_store.cc:276] 0626 08:23:40.379511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.518660 (+139149us) service_pool.cc:225] Handling call
0626 08:23:42.968849 (+2450189us) spinlock_profiling.cc:243] Waited 96.3 ms on lock 0x7b540038d454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:42.969023 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221446016}
I20250626 08:23:42.974078 1843 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250626 08:23:42.919327 3860 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:42.919579 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1627) took 2524 ms. Trace:
I20250626 08:23:42.980059 2000 rpcz_store.cc:276] 0626 08:23:40.395203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.404627 (+2009424us) service_pool.cc:225] Handling call
0626 08:23:42.410434 (+ 5807us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:23:42.444910 (+ 34476us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.444938 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:23:42.464096 (+ 19158us) log.cc:844] Serialized 52 byte log entry
0626 08:23:42.764067 (+299971us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:42.764129 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.764395 (+ 266us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:42.919305 (+154910us) raft_consensus.cc:1683] finished
0626 08:23:42.919328 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:42.919514 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":14506,"thread_start_us":190411,"threads_started":1,"wal-append.queue_time_us":259401}
I20250626 08:23:42.982764 2000 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:42.983440 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1620) took 2590 ms. Trace:
I20250626 08:23:42.984146 1995 rpcz_store.cc:276] 0626 08:23:40.393097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.037969 (+1644872us) service_pool.cc:225] Handling call
0626 08:23:42.038252 (+ 283us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.406309 (+368057us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.406341 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:42.410807 (+ 4466us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:42.410854 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.983133 (+572279us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400270454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:42.983197 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:42.983360 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":846008704}
I20250626 08:23:42.874528 3920 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.983603 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1931) took 2401 ms. Trace:
I20250626 08:23:42.986802 2000 rpcz_store.cc:276] 0626 08:23:40.582667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.982099 (+2399432us) service_pool.cc:225] Handling call
0626 08:23:42.982501 (+ 402us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.983260 (+ 759us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.983486 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:42.907912 3714 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:42.869196 4053 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.949667 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:42.959722 1816 leader_election.cc:336] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 2.212s (SENT)
I20250626 08:23:42.892652 4055 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:42.903877 3727 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:42.995954 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 46359.8us
I20250626 08:23:42.885840 3965 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:42.999243 3965 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:42.963183 1926 raft_consensus.cc:2408] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:42.949304 1965 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:42.871971 3994 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:42.993796 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:42.994261 4008 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:43.003453 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:42.990373 4014 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:42.955857 3908 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:42.923897 1850 raft_consensus.cc:2374] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3. Re-sending same reply.
I20250626 08:23:43.012465 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:43.013286 1843 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:43.014176 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:43.015479 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:43.006740 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.017452 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.011373 1849 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:43.009272 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 60036.4us
I20250626 08:23:43.019408 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:43.021822 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:42.996163 3988 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.002343 4051 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:43.022621 1848 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:43.062872 1843 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:43.027446 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:43.059927 1841 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:43.013211 4029 consensus_meta.cc:287] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.640s user 0.004s sys 0.006s
W20250626 08:23:43.064384 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 49041.9us
I20250626 08:23:43.023278 1968 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.074028 1926 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:43.018095 3994 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:43.075722 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.010970 3860 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:43.018095 3738 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:43.077126 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 195890us
W20250626 08:23:43.077667 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:43.061292 4049 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
I20250626 08:23:43.022532 3965 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:43.058707 3993 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250626 08:23:43.089756 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 70404.3us
I20250626 08:23:43.092244 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:43.092784 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:43.002628 4055 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.096688 2586 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:43.010459 3957 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:43.082130 3861 consensus_meta.cc:287] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.521s user 0.007s sys 0.004s
I20250626 08:23:42.961381 4003 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250626 08:23:43.066380 4051 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:43.090646 3988 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:43.107213 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:43.109246 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 86032.5us
I20250626 08:23:43.011193 3817 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.118468 1968 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:43.123263 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.129050 1968 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:43.124651 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 46795.9us
I20250626 08:23:43.127614 3920 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.093284 3994 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:43.132812 2582 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:43.133491 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.019s sys 0.011s
I20250626 08:23:43.133726 1968 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:43.136874 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.110821 3861 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:43.116529 4051 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.137948 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:43.140758 1814 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:43.141090 4065 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20250626 08:23:43.106915 4062 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:43.144472 4055 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.145437 4053 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.151911 3847 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.153470 3918 raft_consensus.cc:3053] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:23:43.157011 4024 consensus_meta.cc:287] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.087s user 0.012s sys 0.000s
I20250626 08:23:43.159291 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1634) took 1105 ms. Trace:
I20250626 08:23:43.174979 2002 rpcz_store.cc:276] 0626 08:23:42.053849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.092516 (+ 38667us) service_pool.cc:225] Handling call
0626 08:23:43.159119 (+1066603us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b540007a8d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:43.159227 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31887232}
W20250626 08:23:43.161576 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.162003 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1787) took 2584 ms. Trace:
I20250626 08:23:43.163194 2551 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:43.142747 4060 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20250626 08:23:43.166982 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:43.171533 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:43.137763 3988 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:43.170778 3861 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:43.168087 4001 consensus_meta.cc:287] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.583s user 0.011s sys 0.000s
I20250626 08:23:43.176939 1925 rpcz_store.cc:276] 0626 08:23:40.577323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.051742 (+474419us) service_pool.cc:225] Handling call
0626 08:23:41.052094 (+ 352us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:41.825806 (+773712us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:41.825836 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:41.830339 (+ 4503us) log.cc:844] Serialized 26 byte log entry
0626 08:23:41.930757 (+100418us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:41.930805 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:43.157592 (+1226787us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b5400368dd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:43.157639 (+ 47us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:43.157687 (+ 48us) raft_consensus.cc:1683] finished
0626 08:23:43.157700 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:43.161936 (+ 4236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4412,"raft.queue_time_us":53432,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":74402,"spinlock_wait_cycles":1228912128,"thread_start_us":290907,"threads_started":2,"wal-append.queue_time_us":265720}
I20250626 08:23:43.170085 3844 consensus_queue.cc:1046] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20250626 08:23:43.139786 3934 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:43.187285 4014 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:43.227393 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.187621 3970 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:43.188249 3784 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:43.188064 4078 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250626 08:23:43.170665 3791 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:43.173571 3942 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.188977 3981 raft_consensus.cc:3053] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:43.189420 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:43.190027 4058 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:43.191025 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:43.192420 4047 log.cc:927] Time spent T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.481s user 0.001s sys 0.000s
W20250626 08:23:43.194191 4046 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.483s user 0.002s sys 0.000s
W20250626 08:23:43.195704 4045 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.472s user 0.000s sys 0.001s
I20250626 08:23:43.197125 3957 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.198087 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:43.201261 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.204140 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.202997 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:43.211984 3994 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:43.212199 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:43.186167 4024 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.213724 2582 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:43.216887 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1614) took 2864 ms. Trace:
I20250626 08:23:43.218152 4001 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.174604 3918 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:43.502354 4011 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:43.502509 3918 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.206319 4057 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:43.199743 3988 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:43.504660 3988 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:43.222316 3817 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.166096 4003 consensus_queue.cc:1046] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
W20250626 08:23:43.201545 4040 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 1.044s user 0.006s sys 0.006s
W20250626 08:23:43.204959 4027 consensus_meta.cc:287] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.574s user 0.008s sys 0.003s
I20250626 08:23:43.232923 1919 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:43.522461 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 386) took 2249 ms. Trace:
W20250626 08:23:43.317118 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:43.322376 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 90200.6us
W20250626 08:23:43.233811 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.525660 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.244427 3886 raft_consensus.cc:3053] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:43.493013 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:43.495458 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 294272us
I20250626 08:23:43.252743 3722 raft_consensus.cc:3053] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:43.261217 3844 consensus_queue.cc:1046] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.239s
I20250626 08:23:43.520833 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:43.296821 4014 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:43.522946 1919 rpcz_store.cc:276] 0626 08:23:41.272976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.273130 (+ 154us) service_pool.cc:225] Handling call
0626 08:23:43.232478 (+1959348us) spinlock_profiling.cc:243] Waited 8.75 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:43.232762 (+ 284us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:43.522216 (+289454us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:43.522406 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20113664}
W20250626 08:23:43.275969 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 48653.6us
I20250626 08:23:43.532394 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:43.571043 1924 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:23:43.585211 1968 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.531692 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:43.544512 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.566452 2001 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:43.430092 3985 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:43.584095 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 51809.7us
I20250626 08:23:43.581560 2002 raft_consensus.cc:2391] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:43.491560 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 288620us
I20250626 08:23:43.587132 1926 raft_consensus.cc:2391] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:43.591962 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:43.573734 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.548257 1844 rpcz_store.cc:276] 0626 08:23:40.352264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.352942 (+ 678us) service_pool.cc:225] Handling call
0626 08:23:41.635705 (+1282763us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:41.682975 (+ 47270us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:41.683029 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:41.683282 (+ 253us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:41.683346 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:41.683462 (+ 116us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:43.216711 (+1533249us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002f30d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:43.216826 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":40108,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":18501,"spinlock_wait_cycles":2949330816}
I20250626 08:23:43.590618 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:43.602201 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.603605 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.604382 2002 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:43.519995 4083 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:43.605873 1926 raft_consensus.cc:2408] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:23:43.609186 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.605716 1923 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:43.613840 4083 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:43.608635 4057 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250626 08:23:43.614365 4083 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.564874 3722 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:43.614933 4071 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:43.602846 3738 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:43.529011 3981 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:43.615530 4071 raft_consensus.cc:695] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:43.615828 3722 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.616611 4071 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.618676 1967 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:43.528296 3759 raft_consensus.cc:3053] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:43.529419 3886 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:43.628796 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:43.630509 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45388.9us
I20250626 08:23:43.260565 4081 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.364s
I20250626 08:23:43.642092 3981 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.632318 4014 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:43.643467 4081 raft_consensus.cc:3053] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:43.643532 3981 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:43.644017 4081 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:43.253449 4062 consensus_queue.cc:1035] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.359s
W20250626 08:23:43.610052 3961 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250626 08:23:43.655575 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.634709 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:43.656275 1919 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:43.490684 3727 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.643314 4077 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:43.217825 4019 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:43.657490 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.617365 3722 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:43.631433 3759 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:43.668233 1845 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:43.669085 4034 log.cc:927] Time spent T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 1.634s user 0.007s sys 0.026s
I20250626 08:23:43.668936 4014 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:43.526994 4027 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.670339 3857 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.670424 1844 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:43.644893 4081 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:43.672087 2587 raft_consensus.cc:2366] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:43.634171 3738 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:43.674942 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:43.693940 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:43.659106 4031 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.760s user 0.010s sys 0.001s
I20250626 08:23:43.606773 3985 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.678018 1926 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:43.690690 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 393) took 1664 ms. Trace:
I20250626 08:23:43.696262 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:43.661451 4077 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:43.698499 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:43.707578 1849 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:43.696441 1846 rpcz_store.cc:276] 0626 08:23:42.026543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.027310 (+ 767us) service_pool.cc:225] Handling call
0626 08:23:42.042274 (+ 14964us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:42.042635 (+ 361us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.042662 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:42.057755 (+ 15093us) log.cc:844] Serialized 26 byte log entry
0626 08:23:42.373470 (+315715us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:42.438198 (+ 64728us) log.cc:844] Serialized 14 byte log entry
0626 08:23:42.442918 (+ 4720us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:42.443019 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:43.194994 (+751975us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:43.672030 (+477036us) raft_consensus.cc:1683] finished
0626 08:23:43.690482 (+ 18452us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:43.690636 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4375,"prepare.queue_time_us":58736,"prepare.run_cpu_time_us":2526,"prepare.run_wall_time_us":7394,"raft.queue_time_us":15290,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":57210,"spinlock_wait_cycles":21067008,"thread_start_us":283121,"threads_started":1,"wal-append.queue_time_us":489769}
I20250626 08:23:43.712750 1919 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250626 08:23:43.714576 1919 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250626 08:23:43.705384 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:43.719489 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:43.672099 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1607) took 3398 ms (client timeout 2599 ms). Trace:
I20250626 08:23:43.722574 1921 raft_consensus.cc:2366] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3.
I20250626 08:23:43.689688 4014 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:43.724195 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:43.692584 3857 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:43.731151 1844 raft_consensus.cc:2366] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:43.706784 4081 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:43.738364 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:43.739152 1845 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.710058 4077 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:43.709967 4063 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:43.743652 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 214024us
I20250626 08:23:43.730726 2578 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:43.730279 4014 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.698405 4031 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:43.748431 3970 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.507s user 0.006s sys 0.005s
I20250626 08:23:43.776350 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:43.781064 1919 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 4
I20250626 08:23:43.693636 3886 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.732299 3920 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20250626 08:23:43.723613 4086 consensus_queue.cc:1046] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20250626 08:23:43.732128 4062 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20250626 08:23:43.806708 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:43.777129 4063 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.735288 3857 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.774413 3942 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:43.726920 1994 rpcz_store.cc:269] 0626 08:23:40.268491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.269761 (+ 1270us) service_pool.cc:225] Handling call
0626 08:23:43.652948 (+3383187us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54000f5854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:43.667295 (+ 14347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1742100736}
I20250626 08:23:43.782912 3994 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:43.740674 3759 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.830590 1919 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:43.796700 3986 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:43.856956 1845 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:43.776661 4024 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.783385 4001 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.841013 2550 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:43.783666 3970 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:43.831780 3727 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:43.883669 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 125615us
I20250626 08:23:43.858918 3886 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:43.885815 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 106434us
I20250626 08:23:43.872444 3759 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:43.891407 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:43.892297 2582 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:43.894726 2584 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEARNER]: Advancing to term 3
I20250626 08:23:43.882344 3942 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:43.880697 4019 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:43.842151 4007 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20250626 08:23:43.885202 4001 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:43.842326 3994 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:43.906558 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 100726us
I20250626 08:23:43.889551 3920 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:43.878912 3988 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.869779 3986 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.887336 4094 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:43.896368 3985 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.907616 1919 raft_consensus.cc:1238] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:43.896536 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:43.903406 4027 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:43.908998 3942 raft_consensus.cc:695] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:43.921044 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:43.923339 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:43.920848 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:43.929896 1919 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:43.931172 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:43.922231 3934 consensus_meta.cc:287] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.692s user 0.011s sys 0.000s
I20250626 08:23:43.932159 1814 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:43.941802 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:43.942428 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.944212 1968 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:43.965072 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.970382 3936 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:43.971014 3936 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:43.972934 3936 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:43.917227 3920 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:43.974388 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:43.913954 4001 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:43.981694 1849 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:43.995949 3986 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.929237 1994 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:23:43.998150 1998 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1933) took 3414 ms (client timeout 1946 ms). Trace:
W20250626 08:23:43.998714 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 70017.9us
I20250626 08:23:43.945734 3994 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:43.960737 3934 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:43.933676 4031 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:43.999228 1998 rpcz_store.cc:269] 0626 08:23:40.583502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.997966 (+3414464us) service_pool.cc:212] Skipping call since client already timed out
0626 08:23:43.998085 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:23:44.004760 1998 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:44.003420 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:43.995965 4097 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250626 08:23:43.924181 4094 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.012032 1965 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.012504 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1949) took 2957 ms. Trace:
I20250626 08:23:44.004422 4102 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:43.982779 4111 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:44.002465 3857 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.012902 1998 rpcz_store.cc:276] 0626 08:23:41.047942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.004243 (+2956301us) service_pool.cc:225] Handling call
0626 08:23:44.004565 (+ 322us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.005242 (+ 677us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.005463 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:44.096340 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.288807 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:44.289502 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.289947 2586 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 2: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:23:44.170928 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.214991 2582 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
W20250626 08:23:44.096199 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 84207.2us
W20250626 08:23:44.292805 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 348679us
W20250626 08:23:44.029529 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 97434.2us
I20250626 08:23:44.293609 1968 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:44.049437 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.295408 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.297451 2582 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
W20250626 08:23:44.299544 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.299373 2584 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20250626 08:23:44.299926 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:44.300212 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:44.300308 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.303272 1968 leader_election.cc:400] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:44.303553 1968 leader_election.cc:403] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250626 08:23:44.238273 3784 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.999s user 0.004s sys 0.008s
W20250626 08:23:44.132647 4098 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
W20250626 08:23:44.149214 3978 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 2.542s user 0.008s sys 0.058s
I20250626 08:23:44.306371 1998 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:44.014643 4102 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.095978 4104 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.004976 3936 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:44.237896 4095 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.327s user 0.001s sys 0.000s
I20250626 08:23:44.078190 4109 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.010651 3994 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.292407 4027 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.057926 4063 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.093109 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:44.312238 1925 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:44.316220 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1635) took 2262 ms. Trace:
I20250626 08:23:44.334406 1998 rpcz_store.cc:276] 0626 08:23:42.054174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.289011 (+2234837us) service_pool.cc:225] Handling call
0626 08:23:44.306194 (+ 17183us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.311142 (+ 4948us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.316126 (+ 4984us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:44.307109 4001 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:43.998562 4107 consensus_queue.cc:1046] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.325s
I20250626 08:23:44.302522 4111 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:44.327827 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:44.169662 4077 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.304837 4113 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.305058 3934 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:44.311074 4051 consensus_meta.cc:287] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.138s user 0.008s sys 0.010s
I20250626 08:23:44.310278 1968 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.357856 4107 raft_consensus.cc:3053] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:44.358377 4107 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:44.358327 4051 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.309361 3920 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.359563 3920 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:44.359454 4107 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.360067 3920 raft_consensus.cc:695] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:44.360576 4051 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.360692 4107 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.336042 4094 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.360914 3920 consensus_queue.cc:237] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.338495 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1643) took 1320 ms. Trace:
I20250626 08:23:44.330554 3936 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.341430 4027 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.361891 2585 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:44.013427 3942 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.310634 4110 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.324147 4102 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.356485 4091 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:44.358803 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 48613.9us
I20250626 08:23:44.361608 4119 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.337780 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:44.324147 3978 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 2.731s user 0.023s sys 0.077s
I20250626 08:23:44.362073 1998 rpcz_store.cc:276] 0626 08:23:43.017973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.335186 (+1317213us) service_pool.cc:225] Handling call
0626 08:23:44.337804 (+ 2618us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.338040 (+ 236us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:44.338071 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:44.338112 (+ 41us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:44.338144 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.338198 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:44.338440 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:44.385519 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.430464 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:44.432168 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 388649us
W20250626 08:23:44.433892 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.388763 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:44.436214 4111 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:44.370580 3951 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250626 08:23:44.449738 1845 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:44.450984 1848 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:44.452845 1890 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:44.349516 3988 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.454357 4119 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.445880 4091 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.460175 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.363605 3936 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.464504 4067 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.465133 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.475723 1841 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:44.476647 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.477356 1841 raft_consensus.cc:2391] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:44.477211 4077 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.479161 4102 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.365581 4113 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:44.468549 4106 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.355s user 0.001s sys 0.000s
I20250626 08:23:44.459486 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.471841 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.480801 1843 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:44.482354 4091 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:44.457840 3722 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.780s user 0.010s sys 0.000s
I20250626 08:23:44.466931 4119 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.484439 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.493932 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:44.510368 1848 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.333899 4109 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:44.474846 4083 consensus_meta.cc:287] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.860s user 0.012s sys 0.000s
I20250626 08:23:44.511922 1968 leader_election.cc:400] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.515388 1968 leader_election.cc:403] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250626 08:23:44.510175 1841 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:44.516942 1843 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:44.518579 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.518342 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:44.521253 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 47221.2us
I20250626 08:23:44.505367 4077 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.449591 4111 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:44.522485 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:44.513159 4051 consensus_queue.cc:1046] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20250626 08:23:44.491948 4067 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.523391 3988 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:44.524677 3936 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:44.538501 4114 consensus_queue.cc:1046] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
W20250626 08:23:44.540330 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:44.536994 3886 consensus_meta.cc:287] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.646s user 0.011s sys 0.000s
I20250626 08:23:44.520077 4083 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.541591 4124 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.554777 1849 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:23:44.545569 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:44.543800 4108 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.343s user 0.001s sys 0.001s
I20250626 08:23:44.537585 4058 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.541474 3936 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.531623 3722 consensus_queue.cc:1046] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
W20250626 08:23:44.561647 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1950) took 3357 ms (client timeout 2080 ms). Trace:
W20250626 08:23:44.570166 1999 rpcz_store.cc:269] 0626 08:23:41.204531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.838140 (+633609us) service_pool.cc:225] Handling call
0626 08:23:44.561431 (+2723291us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b5400131ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.561576 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306931968}
I20250626 08:23:44.571862 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.564579 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.574208 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:44.561283 1928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1717) took 3138 ms (client timeout 2145 ms). Trace:
W20250626 08:23:44.575688 1928 rpcz_store.cc:269] 0626 08:23:41.422994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.584366 (+161372us) service_pool.cc:225] Handling call
0626 08:23:44.561041 (+2976675us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54001e2154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.561197 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1008942976}
I20250626 08:23:44.543730 4117 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election won for term 6
W20250626 08:23:44.586146 1816 leader_election.cc:336] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.524s (SENT)
I20250626 08:23:44.562106 4083 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.586715 1816 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.587366 4102 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.587978 4102 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.562951 4111 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.598168 4027 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.560494 4058 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.601328 4058 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:44.583760 3738 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.885s user 0.009s sys 0.003s
I20250626 08:23:44.586242 4117 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:44.562916 2587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 590) took 2198 ms (client timeout 1720 ms). Trace:
W20250626 08:23:44.602124 2587 rpcz_store.cc:269] 0626 08:23:42.364309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.366234 (+ 1925us) service_pool.cc:225] Handling call
0626 08:23:44.562675 (+2196441us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003c0354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.562809 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2912980864}
I20250626 08:23:44.563815 4124 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.602742 4117 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.584833 4109 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:44.604241 1816 leader_election.cc:336] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.614s (SENT)
I20250626 08:23:44.604712 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.605865 3952 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.606051 3738 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.577564 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.606839 1994 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:44.607178 1928 raft_consensus.cc:2391] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:44.608592 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 508) took 1828 ms. Trace:
I20250626 08:23:44.608865 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.608974 2583 rpcz_store.cc:276] 0626 08:23:42.779975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.780100 (+ 125us) service_pool.cc:225] Handling call
0626 08:23:42.788587 (+ 8487us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.801093 (+ 12506us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.608399 (+1807306us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x7b5400541fd0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.608528 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18752384}
I20250626 08:23:44.610028 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.611138 4113 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.563598 3714 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.613265 3918 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:44.612871 3714 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.614843 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1641) took 1598 ms. Trace:
I20250626 08:23:44.615249 2002 rpcz_store.cc:276] 0626 08:23:43.016602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.590367 (+573765us) service_pool.cc:225] Handling call
0626 08:23:44.614671 (+1024304us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b540039d354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.614783 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3920640}
W20250626 08:23:43.872241 4087 consensus_queue.cc:1162] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250626 08:23:44.616438 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.618721 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.564358 3722 raft_consensus.cc:3053] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:44.620203 2002 raft_consensus.cc:2391] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:44.620560 1997 raft_consensus.cc:1238] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:44.578406 3817 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.068s user 0.011s sys 0.002s
I20250626 08:23:44.621306 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1622) took 4227 ms. Trace:
I20250626 08:23:44.621381 3986 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.621683 1997 rpcz_store.cc:276] 0626 08:23:40.393661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.379337 (+1985676us) service_pool.cc:225] Handling call
0626 08:23:44.620196 (+2240859us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:44.620389 (+ 193us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.621056 (+ 667us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.621250 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16447232}
I20250626 08:23:44.622004 3986 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.623128 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.624248 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.624851 1997 raft_consensus.cc:2374] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3. Re-sending same reply.
W20250626 08:23:44.627375 4081 consensus_meta.cc:287] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.884s user 0.004s sys 0.008s
I20250626 08:23:44.629447 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.620497 3722 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:44.560663 3936 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.636811 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.634708 1998 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:44.625643 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.635840 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:44.636709 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.641549 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.645097 1997 raft_consensus.cc:2391] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:44.615638 3714 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.646387 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.566479 4067 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.559846 3920 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.658731 3920 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:44.661144 3821 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.128s user 0.010s sys 0.020s
W20250626 08:23:44.661525 3821 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.137s user 0.010s sys 0.021s
I20250626 08:23:44.661792 3920 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.630998 3986 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.651167 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.651484 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.652302 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.650264 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.657572 3988 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.605396 4124 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.639257 4113 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:44.655591 4014 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.872s user 0.005s sys 0.006s
I20250626 08:23:44.662957 1928 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:44.673110 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.663959 2002 raft_consensus.cc:2391] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:44.673431 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.640234 4027 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.667611 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.671331 1999 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:23:44.673332 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:44.663069 1848 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:44.675994 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.650635 3918 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.675345 1997 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 1.
I20250626 08:23:44.677430 1968 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:44.677829 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.680230 1967 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.682565 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:44.687383 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.688433 1997 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:44.689195 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.678750 4124 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:44.702652 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.707051 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.704064 1928 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.704841 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:44.708907 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.707708 1848 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:23:44.710080 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.711289 1968 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:44.709223 2002 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:44.712653 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.713841 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.716077 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.680275 3821 raft_consensus.cc:3053] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:44.718087 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.716970 1928 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:44.714912 2002 raft_consensus.cc:2374] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3. Re-sending same reply.
I20250626 08:23:44.722131 2550 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:44.719570 1967 leader_election.cc:400] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.721825 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.723258 1928 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:44.706382 3918 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:44.707896 4124 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:44.707120 3988 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.732532 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.677117 4014 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.736137 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.717653 3986 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.736888 1816 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.679533 4027 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.733696 3817 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:44.720062 3821 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:44.746583 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.742661 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.737167 1850 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:44.706959 3722 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.734342 3918 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:44.750345 3918 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:44.743599 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 596) took 1542 ms. Trace:
I20250626 08:23:44.751121 2585 rpcz_store.cc:276] 0626 08:23:43.200741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.200862 (+ 121us) service_pool.cc:225] Handling call
0626 08:23:43.201146 (+ 284us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.366281 (+1165135us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.743389 (+377108us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540044e3d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.743531 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2669027712}
I20250626 08:23:44.752166 3759 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:44.752794 3759 raft_consensus.cc:695] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:44.733884 4083 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.753533 4083 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.753619 3759 consensus_queue.cc:237] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.748070 3817 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:44.754411 3936 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.755020 3817 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:44.755807 4083 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.734078 4111 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:44.756515 4111 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.719269 3920 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:44.730612 3738 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.735699 3994 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:44.758853 3738 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.759104 3994 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.760504 4014 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.761070 3994 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.761238 1996 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
I20250626 08:23:44.758133 3920 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.763437 3920 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:44.748175 3986 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.767378 3821 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.768021 3986 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.739825 4124 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:44.768442 3821 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:44.744550 4110 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:44.770252 4117 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:44.767414 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.770859 4117 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.766165 4067 consensus_queue.cc:1046] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250626 08:23:44.773011 4117 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.743664 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 391) took 3187 ms. Trace:
W20250626 08:23:44.773515 1921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1790) took 2722 ms (client timeout 2099 ms). Trace:
I20250626 08:23:44.773959 4081 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:44.774015 1921 rpcz_store.cc:269] 0626 08:23:42.051562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.371638 (+320076us) service_pool.cc:225] Handling call
0626 08:23:44.773313 (+2401675us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400312ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.773444 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3017165056}
W20250626 08:23:44.775496 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.775156 4081 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:44.776078 4111 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.776340 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1788) took 3353 ms. Trace:
I20250626 08:23:44.777082 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.777258 4081 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:44.778224 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.778992 1845 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:44.776110 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:44.782919 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1721) took 2208 ms. Trace:
I20250626 08:23:44.783063 1848 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:44.783496 4113 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.783303 1924 rpcz_store.cc:276] 0626 08:23:42.574112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.621559 (+ 47447us) service_pool.cc:225] Handling call
0626 08:23:42.621864 (+ 305us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:43.585155 (+963291us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.782689 (+1197534us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b54001a9d54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.782864 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2182317952}
I20250626 08:23:44.784853 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.786798 2551 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:44.787976 3813 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:44.788280 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1636) took 4434 ms. Trace:
I20250626 08:23:44.788468 3813 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:44.788671 1842 rpcz_store.cc:276] 0626 08:23:40.353645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:40.886898 (+533253us) service_pool.cc:225] Handling call
0626 08:23:40.916177 (+ 29279us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:41.978313 (+1062136us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.788052 (+2809739us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540038c554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.788225 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2440739072}
I20250626 08:23:44.776728 1918 rpcz_store.cc:276] 0626 08:23:41.423303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.875638 (+452335us) service_pool.cc:225] Handling call
0626 08:23:41.910562 (+ 34924us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.410244 (+499682us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.410279 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:42.410330 (+ 51us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:42.410369 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.776142 (+2365773us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54001aa254. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.776186 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:44.776290 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2498,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":29912,"spinlock_wait_cycles":1138696832}
I20250626 08:23:44.789592 3813 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:44.790946 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.791416 1998 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:44.788564 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:44.792268 1841 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:44.792325 1846 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:44.749410 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.796856 2002 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:44.773674 1927 rpcz_store.cc:276] 0626 08:23:41.556282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.878330 (+322048us) service_pool.cc:225] Handling call
0626 08:23:41.878556 (+ 226us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.380369 (+501813us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:42.380411 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:42.380468 (+ 57us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:42.380506 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.743422 (+2362916us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b540030f054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.743473 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:44.743595 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1153590016}
I20250626 08:23:44.798202 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.798906 1843 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:44.775321 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.800359 1968 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:44.797892 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 48543.3us
I20250626 08:23:44.791646 1918 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.802495 1997 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:44.803244 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.803659 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.804251 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.804381 1997 raft_consensus.cc:2374] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3. Re-sending same reply.
I20250626 08:23:44.806252 1967 leader_election.cc:400] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7.
I20250626 08:23:44.807850 1841 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:44.802183 1816 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:23:44.810132 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.812601 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:44.812763 1816 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.813947 1848 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:44.816044 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.818881 4077 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:44.819413 4077 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.819836 4077 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:44.819908 1847 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1638) took 2660 ms (client timeout 2151 ms). Trace:
W20250626 08:23:44.820389 1847 rpcz_store.cc:269] 0626 08:23:42.159412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.597289 (+437877us) service_pool.cc:225] Handling call
0626 08:23:44.819710 (+2222421us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54001f1154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.819832 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171375360}
I20250626 08:23:44.821373 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.821799 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1632) took 4425 ms. Trace:
I20250626 08:23:44.822150 2001 rpcz_store.cc:276] 0626 08:23:40.396707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.537601 (+2140894us) service_pool.cc:225] Handling call
0626 08:23:42.538117 (+ 516us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:43.586258 (+1048141us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.821568 (+1235310us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400271354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.821742 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2364374272}
I20250626 08:23:44.822543 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.823144 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.823570 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.823827 1842 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:44.824018 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.756547 3936 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.824585 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.825237 2001 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:44.826155 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.827257 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.827822 1997 raft_consensus.cc:2391] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:44.828946 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:44.829849 1999 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:44.835229 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.838145 1921 raft_consensus.cc:2366] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3.
I20250626 08:23:44.760974 3738 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:44.826107 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:44.845170 1816 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.837630 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.854172 1918 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:44.842765 1814 leader_election.cc:400] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3.
I20250626 08:23:44.855125 1814 leader_election.cc:403] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250626 08:23:44.843575 2551 leader_election.cc:336] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 2.004s (SENT)
W20250626 08:23:44.855597 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.855463 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:44.855870 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.809072 3791 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:23:44.844969 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:44.848264 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.858559 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.838376 1846 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:44.849556 3844 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.837718 4081 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:44.858922 1999 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:44.837836 4110 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:44.859973 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:44.853508 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.863241 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.859206 1921 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:44.865164 1967 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:44.845796 4122 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:44.863073 4110 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.868808 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.871238 1921 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:44.849682 4113 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:44.873602 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.849818 4072 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:44.812256 4111 consensus_queue.cc:1046] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.048s
I20250626 08:23:44.872233 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.877194 1997 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:44.872473 2550 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:44.875609 1816 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.859216 3952 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3.
I20250626 08:23:44.878978 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.861724 3936 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:44.880462 1999 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:44.881587 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.882287 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:44.863001 4071 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.870831 4110 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:44.886121 2551 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.886788 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.888686 3988 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:44.876181 4014 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:44.878477 4072 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:44.891945 4143 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250626 08:23:44.867980 4122 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:44.869802 4109 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:44.881758 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.901086 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.900430 1921 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:44.907455 4125 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.374s user 0.003s sys 0.000s
I20250626 08:23:44.914535 4067 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:44.883193 4113 raft_consensus.cc:695] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:44.916594 4132 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:44.917021 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 395) took 2988 ms. Trace:
I20250626 08:23:44.963481 1920 rpcz_store.cc:276] 0626 08:23:41.928448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.440312 (+511864us) service_pool.cc:225] Handling call
0626 08:23:42.522618 (+ 82306us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:42.565081 (+ 42463us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.916714 (+2351633us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b54002aeed0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.916938 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114368384}
I20250626 08:23:44.873412 3738 consensus_queue.cc:1046] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
W20250626 08:23:44.927973 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:44.931354 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1629) took 4535 ms. Trace:
I20250626 08:23:44.970852 1996 rpcz_store.cc:276] 0626 08:23:40.395825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.440935 (+2045110us) service_pool.cc:225] Handling call
0626 08:23:44.562180 (+2121245us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.761631 (+199451us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.930452 (+168821us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400131ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.931138 (+ 686us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b5400131ad0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.931288 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5213441152}
I20250626 08:23:44.932126 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1934) took 4348 ms. Trace:
I20250626 08:23:44.934450 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 515) took 1253 ms. Trace:
W20250626 08:23:44.972646 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 97077.9us
I20250626 08:23:44.950603 1927 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:44.973814 1816 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.972325 1995 rpcz_store.cc:276] 0626 08:23:40.583842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.985545 (+2401703us) service_pool.cc:225] Handling call
0626 08:23:42.993357 (+ 7812us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:43.019915 (+ 26558us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:43.019965 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:43.020057 (+ 92us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:43.020112 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.931905 (+1911793us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b54003a5f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:44.931954 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:44.932070 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43427072}
W20250626 08:23:44.951736 1925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1728) took 1989 ms (client timeout 2005 ms). Trace:
I20250626 08:23:44.954802 1923 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:44.956140 1928 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:23:44.975541 1925 rpcz_store.cc:269] 0626 08:23:42.962769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.494754 (+531985us) service_pool.cc:225] Handling call
0626 08:23:44.951478 (+1456724us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b5400002154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.951655 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1806988544}
I20250626 08:23:44.956974 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:44.915872 2551 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:44.976783 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:44.977463 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 61727.9us
I20250626 08:23:44.977703 1928 raft_consensus.cc:2408] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:44.961350 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.978312 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.978776 1997 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:44.973027 2584 rpcz_store.cc:276] 0626 08:23:43.681275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.682795 (+ 1520us) service_pool.cc:225] Handling call
0626 08:23:43.894582 (+211787us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.300197 (+405615us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.934187 (+633990us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400637750. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:44.934373 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485905664}
I20250626 08:23:44.909446 3936 raft_consensus.cc:695] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:44.969981 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:44.965083 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:44.911243 4014 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:44.982580 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:44.983927 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:45.027729 1995 raft_consensus.cc:2391] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:45.027559 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57695.9us
W20250626 08:23:45.027925 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 44061.6us
I20250626 08:23:45.028803 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:45.029212 1816 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:44.969429 1924 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:44.954077 4122 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:45.029819 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:44.938310 4072 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:45.031915 4053 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:45.032224 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:45.012279 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.033380 3990 log.cc:927] Time spent T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 2.317s user 0.011s sys 0.005s
I20250626 08:23:45.034370 4014 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:44.985601 4071 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:44.950812 4109 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.041853 1814 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:45.032352 4053 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:44.988721 1850 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250626 08:23:45.063952 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.122593 1925 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250626 08:23:44.957574 4067 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:45.162400 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 132652us
I20250626 08:23:45.119606 1921 raft_consensus.cc:1238] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:45.123401 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1807) took 1211 ms. Trace:
I20250626 08:23:45.123488 1918 raft_consensus.cc:1238] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:45.158061 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.163519 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.169224 1995 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:45.170817 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 141690us
I20250626 08:23:44.979355 4027 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:44.982054 4030 raft_consensus.cc:3053] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:45.026872 4119 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:45.166465 1925 rpcz_store.cc:276] 0626 08:23:43.911762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.983054 (+1071292us) service_pool.cc:225] Handling call
0626 08:23:45.122289 (+139235us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:45.123101 (+ 812us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.123338 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:45.166646 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1809) took 1254 ms. Trace:
I20250626 08:23:45.242552 1921 rpcz_store.cc:276] 0626 08:23:43.912329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.982139 (+1069810us) service_pool.cc:225] Handling call
0626 08:23:45.119351 (+137212us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.166359 (+ 47008us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.166592 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:45.185259 1967 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:45.245796 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.171734 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:44.986433 4115 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:45.026525 3738 consensus_queue.cc:1230] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer 548099d81e7f46ac9b2f03d9e5e678a6 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250626 08:23:45.214632 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 172909us
I20250626 08:23:45.177387 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:45.026762 4087 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
W20250626 08:23:45.186390 4141 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
W20250626 08:23:45.275815 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.162467 4109 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:45.162447 4053 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:45.167351 4067 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:44.979902 3817 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.118211 3968 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:23:45.295823 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 124188us
I20250626 08:23:45.328176 1816 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:45.291460 1919 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:45.311786 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 134508us
I20250626 08:23:45.121635 4071 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:45.120208 4062 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.186265 4078 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.213323 3886 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:45.214041 4111 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:45.329752 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1806) took 1418 ms. Trace:
I20250626 08:23:45.186309 3821 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.212958 4117 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.329793 2551 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:45.271697 3759 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.219277 4030 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:45.235715 4038 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.262934 4027 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:45.351289 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 227098us
I20250626 08:23:45.271994 3781 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.357074 1919 rpcz_store.cc:276] 0626 08:23:43.911481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.976137 (+ 64656us) service_pool.cc:225] Handling call
0626 08:23:45.259670 (+1283533us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:45.291241 (+ 31571us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.329487 (+ 38246us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.329696 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9814272}
W20250626 08:23:45.348500 4077 consensus_meta.cc:287] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.528s user 0.010s sys 0.001s
W20250626 08:23:45.358179 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.164772 3920 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.235422 3849 raft_consensus.cc:2762] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20250626 08:23:45.346055 4081 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:45.287004 4119 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:45.322824 3952 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:45.322230 3942 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:45.322608 4087 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:45.460011 4087 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.385344 3759 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.463843 3759 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.347177 3844 consensus_queue.cc:1046] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20250626 08:23:45.371413 4115 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:45.360680 4062 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.388458 1919 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:45.342303 4071 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:45.170917 3918 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.356137 3886 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:45.314877 3738 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
W20250626 08:23:45.439394 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.399647 3781 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:45.424341 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:45.456126 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.486690 3844 raft_consensus.cc:3053] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:45.324146 4113 consensus_queue.cc:237] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.367488 4078 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.381480 4111 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:45.490453 4119 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.540611 1999 raft_consensus.cc:2366] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:45.491848 1965 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:45.377760 3821 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.552723 3821 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.392071 4038 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.431311 3988 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:45.387441 3936 consensus_queue.cc:237] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.378031 4117 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.558360 4117 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.394018 4027 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.559190 4062 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.560173 4027 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.533325 3844 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:45.560897 4072 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.561444 3844 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:45.518155 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.565239 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.418586 3849 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.423621 3920 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.519816 4081 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.525729 3952 consensus_queue.cc:1046] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250626 08:23:45.497411 4030 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.536512 3978 consensus_queue.cc:1046] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250626 08:23:45.587299 3978 raft_consensus.cc:3053] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:45.587695 3978 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:45.588708 4078 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.588572 3978 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.525928 3886 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:45.591960 4067 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.542786 4152 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.545017 3918 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.545646 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1810) took 1243 ms. Trace:
I20250626 08:23:45.593269 4109 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.550420 4111 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:45.564617 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.520226 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 185860us
W20250626 08:23:45.565461 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 73733.3us
I20250626 08:23:45.563129 3844 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:45.606312 3866 raft_consensus.cc:3053] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:45.606743 3866 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:45.607607 3866 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.608628 3866 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:45.535866 4124 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.767s user 0.010s sys 0.000s
I20250626 08:23:45.610728 4124 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.612066 4122 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.612775 4124 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.581485 3849 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.613992 4053 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:45.614684 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1633) took 3561 ms. Trace:
I20250626 08:23:45.615006 3849 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.615110 1994 rpcz_store.cc:276] 0626 08:23:42.053532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.013414 (+1959882us) service_pool.cc:225] Handling call
0626 08:23:44.095526 (+ 82112us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.607227 (+511701us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.614435 (+1007208us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54000f5354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.614618 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1175398784}
I20250626 08:23:45.615928 4053 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.616626 1996 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:45.583348 4081 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.596434 4152 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.619844 1846 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 2: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:45.620911 4081 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.584695 4072 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:45.581296 3968 consensus_queue.cc:1046] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250626 08:23:45.623317 3968 raft_consensus.cc:3053] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:45.623699 3968 raft_consensus.cc:738] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:45.623905 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1637) took 3366 ms (client timeout 1720 ms). Trace:
W20250626 08:23:45.624400 1993 rpcz_store.cc:269] 0626 08:23:42.256839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.378895 (+122056us) service_pool.cc:225] Handling call
0626 08:23:45.623538 (+3244643us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54003a6954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:45.623694 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":339546496}
I20250626 08:23:45.626457 4038 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.626731 2000 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:45.626924 3918 consensus_queue.cc:1046] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:23:45.627343 4083 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:45.627755 4083 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:45.627995 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1948) took 4580 ms. Trace:
I20250626 08:23:45.628433 2000 rpcz_store.cc:276] 0626 08:23:41.047642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:42.992292 (+1944650us) service_pool.cc:225] Handling call
0626 08:23:45.626251 (+2633959us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.627167 (+ 916us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.627793 (+ 626us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b54003a6950. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:45.627931 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6055081088}
W20250626 08:23:45.574808 4014 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.515s user 0.006s sys 0.000s
I20250626 08:23:45.399132 4077 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.604781 3950 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:23:45.605461 4067 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.578630 4062 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:45.627863 3918 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:45.594376 1920 rpcz_store.cc:276] 0626 08:23:44.302339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.992184 (+689845us) service_pool.cc:225] Handling call
0626 08:23:44.992642 (+ 458us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:44.992948 (+ 306us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:44.992989 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:45.031641 (+ 38652us) log.cc:844] Serialized 26 byte log entry
0626 08:23:45.351994 (+320353us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:45.352046 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.352132 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:45.545432 (+193300us) raft_consensus.cc:1683] finished
0626 08:23:45.545451 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.545585 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":319899,"threads_started":1,"wal-append.queue_time_us":503386}
I20250626 08:23:45.617758 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 414) took 1157 ms. Trace:
I20250626 08:23:45.631402 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 407) took 1246 ms. Trace:
I20250626 08:23:45.631799 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 403) took 2537 ms. Trace:
I20250626 08:23:45.637523 1925 rpcz_store.cc:276] 0626 08:23:44.460686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.335330 (+874644us) service_pool.cc:225] Handling call
0626 08:23:45.369819 (+ 34489us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.554115 (+184296us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.554146 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.554199 (+ 53us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:45.554232 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.617535 (+ 63303us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540030f554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.617583 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.617703 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":423344000}
I20250626 08:23:45.638551 1923 rpcz_store.cc:276] 0626 08:23:43.094101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.645251 (+551150us) service_pool.cc:225] Handling call
0626 08:23:44.954356 (+1309105us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.976191 (+ 21835us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.631533 (+655342us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400002150. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:45.631708 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3009890176}
I20250626 08:23:45.633211 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1640) took 1203 ms. Trace:
I20250626 08:23:45.641039 1920 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:45.637629 1849 rpcz_store.cc:276] 0626 08:23:44.385076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.385242 (+ 166us) service_pool.cc:225] Handling call
0626 08:23:45.631245 (+1246003us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000348d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:45.631345 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":262766976}
I20250626 08:23:45.598271 3978 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:45.645538 1920 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:45.635087 3950 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:45.644284 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.644981 1998 raft_consensus.cc:1238] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:45.640017 1844 rpcz_store.cc:276] 0626 08:23:44.429632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.432985 (+ 3353us) service_pool.cc:225] Handling call
0626 08:23:44.433412 (+ 427us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:44.478683 (+ 45271us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:44.478722 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:44.479456 (+ 734us) log.cc:844] Serialized 26 byte log entry
0626 08:23:44.525232 (+ 45776us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:44.531743 (+ 6511us) log.cc:844] Serialized 14 byte log entry
0626 08:23:44.541869 (+ 10126us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.632705 (+1090836us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b540032f5d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.632773 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:45.632860 (+ 87us) raft_consensus.cc:1683] finished
0626 08:23:45.632971 (+ 111us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.633140 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32956,"prepare.queue_time_us":8989,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":35576,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":241,"spinlock_wait_cycles":41858944,"thread_start_us":45240,"threads_started":1,"wal-append.queue_time_us":101788,"wal-append.run_cpu_time_us":2114,"wal-append.run_wall_time_us":1043684}
I20250626 08:23:45.620915 4152 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.650665 1923 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:45.652177 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39250 (request call id 1025) took 1739 ms. Trace:
I20250626 08:23:45.661674 1920 rpcz_store.cc:276] 0626 08:23:43.912920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.649586 (+1736666us) service_pool.cc:225] Handling call
0626 08:23:45.652090 (+ 2504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:45.664011 3950 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.664850 3781 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:45.635993 4122 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:45.667204 4077 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.668613 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 413) took 1208 ms. Trace:
I20250626 08:23:45.704084 1921 rpcz_store.cc:276] 0626 08:23:44.460361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.243574 (+783213us) service_pool.cc:225] Handling call
0626 08:23:45.316045 (+ 72471us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.560826 (+244781us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.560856 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.560898 (+ 42us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:45.560947 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.668310 (+107363us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400242054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.668377 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.668544 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562395904}
W20250626 08:23:45.670544 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.675360 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.706393 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.673779 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1812) took 1237 ms. Trace:
W20250626 08:23:45.686950 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.707167 1924 rpcz_store.cc:276] 0626 08:23:44.436048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.030116 (+594068us) service_pool.cc:225] Handling call
0626 08:23:45.030392 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.030596 (+ 204us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:45.475301 (+444705us) log.cc:844] Serialized 14 byte log entry
0626 08:23:45.673440 (+198139us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.673491 (+ 51us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:45.673532 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.673596 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.673723 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":590499,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":9349,"thread_start_us":492733,"threads_started":2}
I20250626 08:23:45.710199 3950 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:45.711597 4110 consensus_meta.cc:287] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.799s user 0.006s sys 0.004s
I20250626 08:23:45.724799 4109 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.724579 4110 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.641888 4014 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.727821 4014 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:45.727964 4110 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:45.723259 3886 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.714028 1993 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:45.730099 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1967) took 1141 ms. Trace:
I20250626 08:23:45.668017 3817 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:45.730532 1993 rpcz_store.cc:276] 0626 08:23:44.589001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.625673 (+1036672us) service_pool.cc:225] Handling call
0626 08:23:45.637024 (+ 11351us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.728975 (+ 91951us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.729717 (+ 742us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b5400270e54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.729937 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93194112}
I20250626 08:23:45.731727 1994 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:45.732046 1993 raft_consensus.cc:1238] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:45.732702 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1990) took 1051 ms. Trace:
I20250626 08:23:45.732961 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1961) took 1264 ms. Trace:
I20250626 08:23:45.733012 1993 rpcz_store.cc:276] 0626 08:23:44.686358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.731550 (+1045192us) service_pool.cc:225] Handling call
0626 08:23:45.731871 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.732445 (+ 574us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.732649 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:45.714131 1746 catalog_manager.cc:5582] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 08:23:45.711862 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.714952 4048 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.917s user 0.016s sys 0.046s
I20250626 08:23:45.735685 3722 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:45.672502 4067 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:45.624869 3968 consensus_queue.cc:260] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:45.635644 4083 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.742894 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:45.768501 2584 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:45.743189 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:45.745365 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 358197us
I20250626 08:23:45.763875 1746 catalog_manager.cc:5582] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:45.772526 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:45.772820 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:45.773609 1920 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
W20250626 08:23:45.774976 1997 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.013s sys 0.014s
W20250626 08:23:45.775389 1997 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.013s sys 0.014s
I20250626 08:23:45.776300 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1957) took 1412 ms. Trace:
I20250626 08:23:45.776903 1997 rpcz_store.cc:276] 0626 08:23:44.364097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.996418 (+632321us) service_pool.cc:225] Handling call
0626 08:23:44.996802 (+ 384us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.775849 (+779047us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.775897 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.775959 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:45.776010 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.776098 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.776242 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":221686,"raft.queue_time_us":433839,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":556838,"threads_started":1}
I20250626 08:23:45.779879 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:45.744670 3851 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:23:45.780902 2587 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:45.744721 3813 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.782936 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:45.783928 1746 catalog_manager.cc:5582] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 08:23:45.785327 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.755488 3817 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:45.792414 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:45.807422 2582 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:45.792842 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:45.808024 1997 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:45.800169 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:45.809442 1746 catalog_manager.cc:5582] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:45.725839 4071 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.733515 1994 rpcz_store.cc:276] 0626 08:23:44.469053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.616341 (+1147288us) service_pool.cc:225] Handling call
0626 08:23:45.618221 (+ 1880us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.732192 (+113971us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.732746 (+ 554us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000d07d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:45.732907 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":260515712}
I20250626 08:23:45.814819 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:45.789958 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:45.838434 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:45.770938 4067 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.791250 4115 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:45.815884 2587 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:45.813130 1746 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:45.806529 3813 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:45.840529 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 316091us
I20250626 08:23:45.806543 3968 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:45.864502 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 337573us
I20250626 08:23:45.810292 4083 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:45.864100 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.008s sys 0.018s
I20250626 08:23:45.812354 3817 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:45.920029 3844 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:45.899528 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:45.925642 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:45.979694 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 136158us
W20250626 08:23:46.087111 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.087841 2587 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
W20250626 08:23:46.088555 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 218912us
I20250626 08:23:46.090152 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.090759 1844 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:46.091454 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1750) took 1463 ms. Trace:
I20250626 08:23:45.928987 2582 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
W20250626 08:23:45.931685 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.000s sys 0.017s
W20250626 08:23:46.092146 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.000s sys 0.018s
I20250626 08:23:45.795544 3851 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:45.933705 1746 catalog_manager.cc:5582] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:45.944118 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 292) took 1109 ms. Trace:
W20250626 08:23:45.947556 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:46.095970 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.091915 1919 rpcz_store.cc:276] 0626 08:23:44.626228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.508245 (+882017us) service_pool.cc:225] Handling call
0626 08:23:45.535864 (+ 27619us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.536135 (+ 271us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:23:45.811272 (+275137us) log.cc:844] Serialized 14 byte log entry
0626 08:23:46.089403 (+278131us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.089474 (+ 71us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:46.089544 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.089629 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.089786 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":85427,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"thread_start_us":507141,"threads_started":2}
I20250626 08:23:45.921175 4038 consensus_queue.cc:1046] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250626 08:23:46.098102 1995 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:45.901948 2578 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:23:45.978797 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48079us
I20250626 08:23:45.979041 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1811) took 1543 ms. Trace:
I20250626 08:23:45.920629 2580 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:46.096206 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:45.979784 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.009s sys 0.018s
W20250626 08:23:46.098429 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 338651us
I20250626 08:23:46.140623 1927 rpcz_store.cc:276] 0626 08:23:44.435712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.029750 (+594038us) service_pool.cc:225] Handling call
0626 08:23:45.167841 (+138091us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.168083 (+ 242us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:45.635407 (+467324us) log.cc:844] Serialized 14 byte log entry
0626 08:23:45.978681 (+343274us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.978741 (+ 60us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:45.978791 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.978858 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.978980 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11751,"prepare.queue_time_us":453248,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":31496,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":108,"thread_start_us":702166,"threads_started":2}
I20250626 08:23:46.124264 1846 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250626 08:23:46.131978 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:46.075455 4163 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.133s user 0.002s sys 0.000s
W20250626 08:23:46.002475 4160 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.152s user 0.001s sys 0.001s
I20250626 08:23:46.146160 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:46.165900 1841 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:46.145682 1816 leader_election.cc:336] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 2.201s (SENT)
I20250626 08:23:46.149883 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.154783 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.157285 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.124073 3973 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:46.163214 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.096994 3668 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:46.172405 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.254969 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.075124 4113 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.056387 4122 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
I20250626 08:23:46.171767 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:46.256536 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:46.186844 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 1.00162e+06us
W20250626 08:23:46.257889 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.285040 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.304222 1923 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:23:46.211107 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45831.4us
I20250626 08:23:46.255007 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:46.305419 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.131148 1994 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:46.306245 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:46.174352 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.169464 2587 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:46.188776 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:46.261309 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:46.170315 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.143198 4115 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20250626 08:23:45.994117 4109 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:46.336350 3817 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:46.336319 4109 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.337226 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 298) took 1523 ms. Trace:
W20250626 08:23:46.337952 4142 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.486s user 0.006s sys 0.021s
W20250626 08:23:46.333397 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.338066 4111 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:46.147296 4014 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:46.339401 4014 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.339882 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:46.143286 3918 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:46.340708 3918 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.341442 4014 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.342672 3918 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.304638 1843 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:46.334146 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.337673 1994 rpcz_store.cc:276] 0626 08:23:44.814063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.843159 (+1029096us) service_pool.cc:225] Handling call
0626 08:23:46.002179 (+159020us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.336947 (+334768us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.337168 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.144098 3994 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.346432 3994 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.347126 4067 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:46.347716 4171 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250626 08:23:46.348695 3994 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.349505 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 296) took 1536 ms. Trace:
I20250626 08:23:46.349864 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.350443 2581 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:23:46.350667 4109 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:46.351405 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:46.352018 1841 raft_consensus.cc:604] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:46.352350 1841 raft_consensus.cc:612] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:23:46.266587 3973 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.345600 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.353933 4014 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 08:23:46.285830 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:46.355428 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 306) took 1515 ms (client timeout 2004 ms). Trace:
I20250626 08:23:46.260710 4071 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:46.355822 1999 rpcz_store.cc:269] 0626 08:23:44.840282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.881127 (+ 40845us) service_pool.cc:225] Handling call
0626 08:23:46.355256 (+1474129us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54003a6454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:46.355374 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1512424192}
I20250626 08:23:46.356940 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.357466 1999 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:46.167409 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:46.085755 4162 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.186s
I20250626 08:23:46.080477 2579 raft_consensus.cc:1238] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:46.332784 2582 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:46.327733 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.334813 1924 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:46.346788 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 1980) took 1754 ms (client timeout 2201 ms). Trace:
W20250626 08:23:46.382128 2002 rpcz_store.cc:269] 0626 08:23:44.592534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.742436 (+149902us) service_pool.cc:225] Handling call
0626 08:23:46.346626 (+1604190us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b5400280ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:46.346735 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112931712}
I20250626 08:23:46.354275 1920 raft_consensus.cc:2408] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:23:46.358002 3973 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:46.357697 3978 consensus_meta.cc:287] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.707s user 0.001s sys 0.009s
I20250626 08:23:46.349890 1997 rpcz_store.cc:276] 0626 08:23:44.813441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.808723 (+995282us) service_pool.cc:225] Handling call
0626 08:23:46.347914 (+539191us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.348130 (+ 216us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.348161 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.348207 (+ 46us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:46.348247 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.348303 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.349348 (+ 1045us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400280ad0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:46.349446 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239347968}
I20250626 08:23:46.381486 1919 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:46.258117 1849 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:46.312283 1893 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:46.238512 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:46.387956 1890 leader_election.cc:336] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Timed out: RequestConsensusVote RPC to 127.0.171.196:37733 timed out after 1.720s (SENT)
I20250626 08:23:46.366971 1927 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:23:46.384449 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.398689 1997 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:46.399646 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.400213 1997 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:46.403396 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 257733us
I20250626 08:23:46.405480 1814 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:46.359349 4168 log.cc:927] Time spent T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250626 08:23:46.350582 2578 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:46.407963 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.408824 1994 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 2: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:46.409366 3866 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.800s user 0.014s sys 0.006s
I20250626 08:23:46.413996 2000 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20250626 08:23:46.414407 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.414408 2000 pending_rounds.cc:85] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 6
I20250626 08:23:46.417425 2000 pending_rounds.cc:107] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20250626 08:23:46.417728 2000 raft_consensus.cc:2887] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:46.266427 3813 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:46.152686 4154 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:46.434197 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.439440 1841 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:46.265429 4167 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.446477 1848 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:46.247107 3886 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.385447 3668 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:46.473771 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:46.256007 4062 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.151s
I20250626 08:23:46.380975 3994 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:46.409678 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.464210 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.465080 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.475306 1999 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:46.465278 3951 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
W20250626 08:23:46.509723 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 365741us
I20250626 08:23:46.467648 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1737) took 2556 ms. Trace:
I20250626 08:23:46.470279 1924 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:46.510450 1926 rpcz_store.cc:276] 0626 08:23:43.911086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:43.932611 (+ 21525us) service_pool.cc:225] Handling call
0626 08:23:43.976511 (+ 43900us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:43.976945 (+ 434us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:43.976985 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:43.977532 (+ 547us) log.cc:844] Serialized 26 byte log entry
0626 08:23:44.791028 (+813496us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b18003955c0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc2f01 00007f4eb9051ccf
0626 08:23:44.857175 (+ 66147us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:44.857212 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:44.857277 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:46.337345 (+1480068us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:46.467380 (+130035us) raft_consensus.cc:1683] finished
0626 08:23:46.467403 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.467578 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":15495,"spinlock_wait_cycles":21200512,"thread_start_us":65980,"threads_started":1,"wal-append.queue_time_us":873974}
W20250626 08:23:46.472157 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 160107us
I20250626 08:23:46.512961 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:46.514068 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:46.515146 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:46.415052 4116 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:46.332345 4113 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.485852 1997 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:46.478485 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.500702 1844 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:46.500780 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.097877 1993 rpcz_store.cc:276] 0626 08:23:44.812376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.767148 (+954772us) service_pool.cc:225] Handling call
0626 08:23:45.810145 (+ 42997us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.839053 (+ 28908us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.850282 (+ 11229us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:45.859715 (+ 9433us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:45.865080 (+ 5365us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.871845 (+ 6765us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:45.921360 (+ 49515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.514094 1816 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:46.564383 1850 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:23:46.504853 1848 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:46.499643 1890 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.565496 1850 raft_consensus.cc:604] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:46.566042 1850 raft_consensus.cc:612] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250626 08:23:46.566623 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 52600.6us
W20250626 08:23:46.505429 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 380752us
I20250626 08:23:46.567018 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:46.567427 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:46.508669 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:46.517315 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.513154 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 299) took 1685 ms. Trace:
I20250626 08:23:46.571662 2002 rpcz_store.cc:276] 0626 08:23:44.827469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.512369 (+1684900us) service_pool.cc:225] Handling call
0626 08:23:46.512675 (+ 306us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.512852 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.512875 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.512915 (+ 40us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:46.512951 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.513007 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.513102 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:46.548703 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.467078 4156 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.546787 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.579638 2583 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:46.523198 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.580489 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:46.580888 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57837.6us
I20250626 08:23:46.581372 2581 raft_consensus.cc:2408] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:23:46.503492 4083 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.406406 3978 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:46.549098 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:46.573753 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.546417 4153 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:46.519973 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.496757 3668 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:46.575295 4162 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.576438 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 301) took 1748 ms. Trace:
I20250626 08:23:46.632334 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:46.466827 4166 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:46.509573 3973 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
W20250626 08:23:46.719208 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.601482 4083 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.588101 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:46.255792 3851 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.574265 1999 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:46.509402 3951 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:46.600857 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:46.574518 1920 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:46.496678 3970 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20250626 08:23:46.573024 3566 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.468163 4154 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.582113 3781 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:46.606704 4172 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20250626 08:23:46.607715 1816 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:46.575954 2002 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:46.613143 3942 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:46.630429 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.507601 3982 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20250626 08:23:46.580201 3886 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:46.632191 1997 rpcz_store.cc:276] 0626 08:23:44.828385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.575669 (+1747284us) service_pool.cc:225] Handling call
0626 08:23:46.575901 (+ 232us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.576064 (+ 163us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.576090 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.576133 (+ 43us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:46.576166 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.576220 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.576366 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.564777 1849 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:46.586966 3968 consensus_meta.cc:287] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.695s user 0.005s sys 0.005s
I20250626 08:23:46.586454 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.631230 3978 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:23:46.640547 4089 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:46.510164 3994 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:46.631700 4150 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20250626 08:23:46.468348 4167 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.607595 4011 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:46.631064 3668 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:46.608291 3813 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:46.719861 4153 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:46.585376 4156 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:46.788340 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:46.722626 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 90374.8us
I20250626 08:23:46.639261 4166 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:46.720875 4161 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20250626 08:23:46.755040 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.762776 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 302) took 1934 ms. Trace:
I20250626 08:23:46.850572 2002 rpcz_store.cc:276] 0626 08:23:44.828727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.575490 (+1746763us) service_pool.cc:225] Handling call
0626 08:23:46.575798 (+ 308us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.762437 (+186639us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.762700 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.765633 1997 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:46.852414 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 311) took 2000 ms. Trace:
I20250626 08:23:46.852739 1997 rpcz_store.cc:276] 0626 08:23:44.852453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.764950 (+1912497us) service_pool.cc:225] Handling call
0626 08:23:46.765278 (+ 328us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.852168 (+ 86890us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.852363 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.854208 1997 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:46.763434 3942 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:46.769644 3886 raft_consensus.cc:695] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:46.856146 3942 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:46.758901 3566 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.856801 3886 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.857053 3942 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:46.769321 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 161695us
I20250626 08:23:46.858261 3566 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.823132 3978 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:46.819918 4011 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:46.859618 4167 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:46.812680 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.860059 4011 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:46.769517 3781 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:46.779608 3951 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:46.797156 3722 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.025s user 0.011s sys 0.000s
W20250626 08:23:46.780107 4175 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20250626 08:23:46.859629 3978 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.778985 4077 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:46.858285 3942 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:46.863591 4113 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:46.863420 4077 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.861430 3994 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:46.769726 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 304) took 1930 ms. Trace:
I20250626 08:23:46.861377 3781 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:46.865615 1997 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:46.865851 2583 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:23:46.757303 4177 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20250626 08:23:46.802881 1999 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 08:23:46.869688 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:46.862856 2002 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:46.870325 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 305) took 2030 ms. Trace:
I20250626 08:23:46.870790 1999 rpcz_store.cc:276] 0626 08:23:44.839967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.730117 (+1890150us) service_pool.cc:225] Handling call
0626 08:23:46.802453 (+ 72336us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.870009 (+ 67556us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.870259 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:46.720754 4162 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.767356 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:46.866655 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1825) took 1938 ms. Trace:
I20250626 08:23:46.742919 4038 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:46.764758 4178 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
W20250626 08:23:46.881312 1965 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.008s sys 0.032s
I20250626 08:23:46.872200 1925 rpcz_store.cc:276] 0626 08:23:44.928415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.639138 (+710723us) service_pool.cc:225] Handling call
0626 08:23:45.648187 (+ 9049us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:45.648398 (+ 211us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.648422 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:45.673342 (+ 24920us) log.cc:844] Serialized 26 byte log entry
0626 08:23:46.145170 (+471828us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:46.145235 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.145324 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:46.866439 (+721115us) raft_consensus.cc:1683] finished
0626 08:23:46.866459 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.866596 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":414120,"threads_started":1,"wal-append.queue_time_us":735006}
W20250626 08:23:46.883019 1965 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.510s user 0.008s sys 0.032s
W20250626 08:23:46.883364 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 510830us
I20250626 08:23:46.741852 1847 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:46.873289 1920 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:46.828019 4153 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:46.861717 3951 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:46.874934 1842 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.093s
I20250626 08:23:46.742547 4122 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.752782 4062 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.839325 3918 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:46.779891 1993 raft_consensus.cc:1238] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:46.869221 4111 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.028s sys 0.047s
I20250626 08:23:46.864574 1994 rpcz_store.cc:276] 0626 08:23:44.839581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.608395 (+1768814us) service_pool.cc:225] Handling call
0626 08:23:46.768940 (+160545us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:46.769153 (+ 213us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.769355 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.769382 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.769434 (+ 52us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:46.769464 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.769535 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.769666 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5207936}
I20250626 08:23:46.865309 3781 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:46.866775 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 367371us
W20250626 08:23:46.928467 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 419891us
I20250626 08:23:46.927278 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:46.929404 1890 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.862713 4083 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:46.795154 4089 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:46.887840 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:46.892774 4153 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:46.866868 3715 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:46.892315 1842 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.000s sys 0.093s
I20250626 08:23:46.850117 4166 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:46.893529 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 379504us
I20250626 08:23:46.943825 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:46.764957 4119 consensus_queue.cc:1046] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20250626 08:23:46.895332 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1951) took 2561 ms. Trace:
W20250626 08:23:46.925529 4111 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.028s sys 0.048s
I20250626 08:23:46.947365 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.902704 1814 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:46.862807 3970 consensus_queue.cc:1162] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250626 08:23:46.840111 3813 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.921845 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 528) took 1146 ms. Trace:
W20250626 08:23:46.953245 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 50671.3us
I20250626 08:23:46.760427 4154 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:46.922464 1965 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.953639 2585 rpcz_store.cc:276] 0626 08:23:45.775591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.775714 (+ 123us) service_pool.cc:225] Handling call
0626 08:23:45.776063 (+ 349us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:45.776430 (+ 367us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:45.776458 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:45.783202 (+ 6744us) log.cc:844] Serialized 26 byte log entry
0626 08:23:46.245173 (+461971us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:46.245243 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.245340 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:46.921618 (+676278us) raft_consensus.cc:1683] finished
0626 08:23:46.921640 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.921775 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13067,"raft.queue_time_us":4129,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":9649,"spinlock_wait_cycles":376576,"thread_start_us":381374,"threads_started":1,"wal-append.queue_time_us":622324}
I20250626 08:23:46.927621 1816 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:46.930395 2551 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:46.930398 1890 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.930931 1968 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:46.892371 4164 consensus_queue.cc:1046] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20250626 08:23:46.919512 3918 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:46.927675 4122 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:46.943902 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 56200.4us
I20250626 08:23:46.926406 4077 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:46.962987 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:46.875398 4056 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.015s sys 0.026s
I20250626 08:23:46.917073 4116 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.917421 4109 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.915812 3936 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:46.925004 4062 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:46.951658 2002 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:46.874059 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.050s sys 0.078s
I20250626 08:23:46.947441 2001 rpcz_store.cc:276] 0626 08:23:44.333678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.826108 (+492430us) service_pool.cc:225] Handling call
0626 08:23:44.826369 (+ 261us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:44.826582 (+ 213us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:44.951056 (+124474us) log.cc:844] Serialized 14 byte log entry
0626 08:23:46.572680 (+1621624us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b1800387220. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc306a
0626 08:23:46.894977 (+322297us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.895035 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:46.895082 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.895150 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:46.895276 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41614,"prepare.queue_time_us":201546,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":78350,"raft.queue_time_us":287364,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":62405888,"thread_start_us":700848,"threads_started":3,"wal-append.queue_time_us":427319,"wal-append.run_cpu_time_us":1306,"wal-append.run_wall_time_us":1034257}
I20250626 08:23:46.934412 4089 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:46.966141 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.944406 4181 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
W20250626 08:23:46.976518 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45700.3us
I20250626 08:23:46.990942 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.991640 2002 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:46.980157 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1958) took 2615 ms. Trace:
I20250626 08:23:46.992561 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:46.992365 1996 rpcz_store.cc:276] 0626 08:23:44.364380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.033272 (+668892us) service_pool.cc:225] Handling call
0626 08:23:45.033590 (+ 318us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.617036 (+583446us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.979812 (+1362776us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54001658d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:46.980084 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1340398336}
W20250626 08:23:46.983945 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.649s user 0.050s sys 0.078s
W20250626 08:23:46.993882 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 659372us
I20250626 08:23:46.926133 4113 consensus_queue.cc:1046] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20250626 08:23:46.968861 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:46.947445 4167 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:46.950109 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 303) took 2118 ms. Trace:
I20250626 08:23:46.949038 3982 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:46.989410 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:46.950739 4111 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:23:46.974529 4056 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.615s user 0.015s sys 0.027s
I20250626 08:23:46.933504 3994 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250626 08:23:46.977308 4087 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:46.976935 3918 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:47.008656 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:46.964210 4183 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:46.974427 3722 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:46.950269 3715 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.027065 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:47.001472 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:47.065057 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 67658.7us
W20250626 08:23:47.061518 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 131160us
W20250626 08:23:47.067025 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:47.067404 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.067853 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:47.068128 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 124406us
I20250626 08:23:47.068460 2582 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 1.
I20250626 08:23:47.069387 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:47.024830 3982 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:46.997160 4119 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.006727 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:47.079474 1893 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:47.080489 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:46.986479 4109 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:46.981726 4116 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.009537 4193 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:47.093799 1920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:23:47.094789 1920 raft_consensus.cc:604] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:46.988188 3936 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.015661 4087 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:47.096164 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.008029 4181 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:47.100436 1924 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:47.105096 1924 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250626 08:23:47.101940 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 107293us
I20250626 08:23:47.109400 2001 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:47.064567 4184 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.070632 3982 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:47.024834 4161 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250626 08:23:47.082648 4189 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:47.085250 4154 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:47.001247 1993 rpcz_store.cc:276] 0626 08:23:44.829108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.582827 (+1753719us) service_pool.cc:225] Handling call
0626 08:23:46.719691 (+136864us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.922537 (+202846us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.947825 (+ 25288us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:47.115854 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:47.116180 2550 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:47.119045 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 08:23:47.086513 1994 raft_consensus.cc:2374] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4. Re-sending same reply.
I20250626 08:23:47.091691 2578 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:47.123728 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2000) took 1015 ms. Trace:
W20250626 08:23:47.172049 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 55950.1us
W20250626 08:23:47.169698 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 90302.1us
I20250626 08:23:47.171849 1997 rpcz_store.cc:276] 0626 08:23:46.108159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.869238 (+761079us) service_pool.cc:225] Handling call
0626 08:23:46.869613 (+ 375us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:46.872536 (+ 2923us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.872561 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:46.883414 (+ 10853us) log.cc:844] Serialized 26 byte log entry
0626 08:23:46.995959 (+112545us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:46.996010 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.996095 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:47.123459 (+127364us) raft_consensus.cc:1683] finished
0626 08:23:47.123479 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:47.123657 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":48940,"threads_started":1,"wal-append.queue_time_us":180691}
I20250626 08:23:47.117852 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.108646 4087 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.173928 1920 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:47.175110 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.176613 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:47.127801 1965 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:47.012050 4167 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.121459 4200 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:47.162168 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:23:47.102047 4195 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:47.178382 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 62865.3us
I20250626 08:23:47.187783 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.116328 4156 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:47.190629 1918 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:47.084079 4119 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:47.192675 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.151561 4189 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:47.194167 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 266610us
I20250626 08:23:47.196434 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:47.197938 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1814) took 2761 ms. Trace:
I20250626 08:23:47.206727 1920 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.206626 1918 rpcz_store.cc:276] 0626 08:23:44.436867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.167138 (+730271us) service_pool.cc:225] Handling call
0626 08:23:45.167462 (+ 324us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:47.190440 (+2022978us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b5400175aa0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900e2f0 00007f4eb9080da0 00007f4eb9081e11 00007f4eb907e97f 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c
0626 08:23:47.197650 (+ 7210us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:47.197858 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8766208}
I20250626 08:23:47.200084 1848 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:47.162029 4190 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.195703 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.207980 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:47.227061 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.227721 1918 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:47.179338 4077 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 08:23:47.151842 4154 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:47.318940 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.323410 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.320122 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.323902 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.321347 1816 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:47.321847 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 120914us
I20250626 08:23:47.324698 1849 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:23:47.325876 1848 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:47.333006 4183 raft_consensus.cc:695] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:47.334688 4183 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.342340 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.343082 1847 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:47.190765 4200 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:47.345139 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.346405 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.398672 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.398895 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.194988 4067 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:47.399540 1850 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:47.400429 2550 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:47.401304 1890 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.196:37733 timed out after 2.000s (SENT) [suppressed 36 similar messages]
I20250626 08:23:47.402041 1848 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:47.411425 1920 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:47.412835 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.414299 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.415572 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.418668 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:47.195129 4071 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.224s
I20250626 08:23:47.321620 4077 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:47.228389 3781 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.421854 4154 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.418851 4195 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.322461 4162 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:47.423818 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:47.471241 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:47.425961 4194 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.238s user 0.000s sys 0.000s
I20250626 08:23:47.124833 4184 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:47.426834 4182 log.cc:927] Time spent T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.321s user 0.000s sys 0.000s
W20250626 08:23:47.427779 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 106533us
I20250626 08:23:47.429510 1920 raft_consensus.cc:2408] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:23:47.429205 4188 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.496089 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.225206 4189 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.439319 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:47.441839 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.495649 4216 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.497799 3950 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:47.498301 3950 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:47.423831 4122 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.116833 4111 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.499233 3950 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.500388 3950 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:47.400648 3813 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.502480 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1960) took 3137 ms. Trace:
I20250626 08:23:47.487452 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.502955 1995 rpcz_store.cc:276] 0626 08:23:44.364978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.170222 (+805244us) service_pool.cc:225] Handling call
0626 08:23:45.218669 (+ 48447us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.098576 (+879907us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:47.502249 (+1403673us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54002c3654. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:47.502422 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2021913344}
I20250626 08:23:47.401432 4200 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.437702 4077 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:47.431944 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 404939us
I20250626 08:23:47.507725 1968 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:47.492707 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:47.422189 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.497900 4216 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.418159 4067 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:47.501708 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.423235 3906 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:47.514688 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.517591 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1834) took 1154 ms. Trace:
I20250626 08:23:47.520943 4164 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.518127 1921 rpcz_store.cc:276] 0626 08:23:46.363559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.367578 (+ 4019us) service_pool.cc:225] Handling call
0626 08:23:46.399046 (+ 31468us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:46.402203 (+ 3157us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.402236 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:46.498197 (+ 95961us) log.cc:844] Serialized 26 byte log entry
0626 08:23:46.951913 (+453716us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:46.952004 (+ 91us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.952264 (+ 260us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:47.517358 (+565094us) raft_consensus.cc:1683] finished
0626 08:23:47.517384 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:47.517530 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2706,"raft.queue_time_us":6616,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":23776,"thread_start_us":343845,"threads_started":1,"wal-append.queue_time_us":553581}
I20250626 08:23:47.522171 1816 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:47.521814 4164 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.524667 3781 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:47.525050 3781 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:47.525163 4164 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.525420 3781 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:47.525982 4164 consensus_queue.cc:1046] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:47.525521 1921 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:47.527278 4164 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:47.472154 4062 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:47.533509 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.534150 1925 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:23:47.534756 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.006s sys 0.000s
W20250626 08:23:47.535202 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.007s sys 0.000s
W20250626 08:23:47.535552 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 559240us
I20250626 08:23:47.493079 4162 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:47.496627 4154 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.319337 4190 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.491812 4203 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:47.548072 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:47.549088 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:47.516145 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:47.513851 4067 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.502504 4116 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.577379 1923 raft_consensus.cc:3053] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:47.633446 1923 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:47.533286 4213 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:47.589798 2551 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:47.495728 4087 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.427774 3982 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.625454 1918 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:47.545967 3968 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20250626 08:23:47.589077 4071 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:47.538312 4122 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20250626 08:23:47.538115 3906 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:47.638954 4207 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20250626 08:23:47.549675 4062 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:47.546600 4183 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:47.629576 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1666) took 1524 ms. Trace:
I20250626 08:23:47.551358 4215 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.400002 4181 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.539811 4162 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:47.638164 4196 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.445s user 0.001s sys 0.000s
I20250626 08:23:47.538573 4209 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20250626 08:23:47.586491 4109 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:47.638955 4167 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:47.471114 4195 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:47.641144 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:47.634461 1923 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.549194 4204 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20250626 08:23:47.511147 4188 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.641744 4203 raft_consensus.cc:3053] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:47.630844 4192 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:47.550702 1847 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:47.638177 4119 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:47.650058 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 60360.7us
W20250626 08:23:47.667459 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 143434us
W20250626 08:23:47.672367 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.664927 1844 raft_consensus.cc:2366] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
W20250626 08:23:47.679256 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.651604 4071 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:47.673949 4132 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
W20250626 08:23:47.658875 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:47.707742 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 306449us
I20250626 08:23:47.678540 1842 rpcz_store.cc:276] 0626 08:23:46.105112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.105583 (+ 471us) service_pool.cc:225] Handling call
0626 08:23:46.146611 (+ 41028us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:46.171205 (+ 24594us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.171247 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:46.171829 (+ 582us) log.cc:844] Serialized 26 byte log entry
0626 08:23:46.934289 (+762460us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:46.934333 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:46.934398 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:47.629314 (+694916us) raft_consensus.cc:1683] finished
0626 08:23:47.629342 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:47.629500 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24842,"raft.queue_time_us":25096,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":42834,"thread_start_us":628501,"threads_started":1,"wal-append.queue_time_us":759857}
W20250626 08:23:47.709523 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:47.689123 4223 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.659415 4219 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:47.707593 4162 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:47.688037 4111 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:47.702353 4215 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:48.132061 1846 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.004s sys 0.013s
W20250626 08:23:48.538228 1846 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.000s user 0.004s sys 0.025s
W20250626 08:23:47.702239 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 168430us
W20250626 08:23:47.915405 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.000s sys 0.015s
I20250626 08:23:47.942167 4190 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:47.944674 4153 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.994s user 0.008s sys 0.001s
I20250626 08:23:48.543143 4153 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.543886 4223 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:48.545523 4153 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:47.906070 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 531588us
W20250626 08:23:48.547384 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:48.550261 1923 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:47.906831 1926 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.006s sys 0.008s
W20250626 08:23:48.551426 1926 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.319s user 0.011s sys 0.023s
I20250626 08:23:48.551605 2582 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:48.551345 1920 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:48.553095 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 429) took 1458 ms. Trace:
I20250626 08:23:48.554124 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:48.554718 2581 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 5
I20250626 08:23:48.554863 1920 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:48.556396 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:48.062640 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.007s sys 0.025s
W20250626 08:23:48.558274 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.018s sys 0.026s
I20250626 08:23:48.558033 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:47.890913 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:48.558990 1997 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:48.001715 1921 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:48.541776 4244 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:48.561185 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1765) took 1435 ms. Trace:
I20250626 08:23:48.561658 1921 rpcz_store.cc:276] 0626 08:23:47.125661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.525181 (+399520us) service_pool.cc:225] Handling call
0626 08:23:47.525421 (+ 240us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:48.560885 (+1035464us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:48.561119 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:48.542474 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.369s user 0.003s sys 0.026s
I20250626 08:23:48.553499 1926 rpcz_store.cc:276] 0626 08:23:47.094723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.094891 (+ 168us) service_pool.cc:225] Handling call
0626 08:23:47.114091 (+ 19200us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:47.154274 (+ 40183us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:47.154322 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:47.170319 (+ 15997us) log.cc:844] Serialized 26 byte log entry
0626 08:23:48.552734 (+1382415us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:48.552773 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:48.552889 (+ 116us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:48.552931 (+ 42us) raft_consensus.cc:1683] finished
0626 08:23:48.552942 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:48.553043 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39764,"raft.queue_time_us":381742,"spinlock_wait_cycles":19735808,"thread_start_us":1320089,"threads_started":1,"wal-append.queue_time_us":1342463}
I20250626 08:23:48.589439 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:48.590305 1997 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
W20250626 08:23:48.564195 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.39072e+06us
W20250626 08:23:47.791208 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 282172us
I20250626 08:23:48.689503 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:48.689987 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:47.711336 4062 raft_consensus.cc:695] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:23:48.042955 3951 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.148s user 0.008s sys 0.004s
W20250626 08:23:48.537990 3950 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.032s user 0.012s sys 0.000s
W20250626 08:23:48.073731 3918 consensus_meta.cc:287] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.010s user 0.007s sys 0.004s
I20250626 08:23:48.740584 1920 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250626 08:23:47.848610 4175 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20250626 08:23:48.108109 4189 consensus_meta.cc:287] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.577s user 0.013s sys 0.006s
W20250626 08:23:48.227635 3668 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.288s user 0.013s sys 0.000s
W20250626 08:23:47.949298 4011 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.089s user 0.008s sys 0.004s
I20250626 08:23:48.850905 4189 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.851226 4011 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.853015 4189 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.853353 2587 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:48.773746 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:48.344679 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.006s sys 0.010s
W20250626 08:23:47.955211 4228 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
W20250626 08:23:48.869608 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.192s user 0.007s sys 0.012s
W20250626 08:23:48.870231 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:48.444614 4077 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.906s user 0.010s sys 0.000s
W20250626 08:23:48.099920 4089 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.099s user 0.011s sys 0.001s
W20250626 08:23:47.979107 4166 consensus_meta.cc:287] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.017s user 0.007s sys 0.004s
W20250626 08:23:47.859095 4218 log.cc:927] Time spent T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20250626 08:23:48.729663 4164 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:48.730337 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:47.948321 3942 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.085s user 0.007s sys 0.002s
I20250626 08:23:48.787858 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:48.373131 4067 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.724s user 0.010s sys 0.002s
I20250626 08:23:47.965227 3781 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:47.724234 4203 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:48.093426 3906 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.855312 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:48.728515 4111 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.964803 4236 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:48.965551 4236 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.968530 4236 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.969787 4254 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:48.970531 4254 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.973104 4254 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.727643 4244 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.981433 4244 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:48.226706 4183 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:49.036931 4183 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.039616 4183 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:47.950376 4087 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.097039 4188 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:48.622165 4071 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:47.749522 4214 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:49.056819 4214 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:47.840054 4184 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:47.647475 2578 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:49.060041 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 535) took 1579 ms. Trace:
I20250626 08:23:49.060485 2578 rpcz_store.cc:276] 0626 08:23:47.480422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.486449 (+ 6027us) service_pool.cc:225] Handling call
0626 08:23:49.059950 (+1573501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:49.061878 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:48.853027 4011 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:48.621050 4197 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:48.691808 4223 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:49.064378 4011 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:49.065043 4011 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.067480 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:49.089422 4188 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.090504 4188 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.854512 3951 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:48.986763 4203 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.093483 3951 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.094604 4203 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:49.096292 3951 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.958568 3906 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:49.179778 3906 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.989259 4215 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.989945 1994 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:23:48.909762 4077 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.185366 4077 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:49.186612 2578 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:49.190127 3906 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.942519 4166 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.197420 4166 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.855929 3918 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.008708 4111 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:48.691483 4219 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.011615 4030 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:48.850958 3668 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.071211 4011 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:49.073081 4241 consensus_queue.cc:1046] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:49.077499 3781 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.082995 4062 consensus_queue.cc:237] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.089512 4223 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:23:49.284428 4011 raft_consensus.cc:3053] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:49.285255 4011 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:49.286643 4011 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:48.727209 3982 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.290880 3982 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.293812 3982 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:49.459492 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.919s user 0.000s sys 0.006s
I20250626 08:23:49.460227 1841 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:49.461898 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1683) took 1949 ms. Trace:
I20250626 08:23:48.799760 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:48.978479 1993 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:48.908090 1995 raft_consensus.cc:2391] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:49.124760 1996 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:49.223057 2002 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.017s sys 0.045s
W20250626 08:23:49.574836 2612 heartbeater.cc:646] Failed to heartbeat to 127.0.171.254:42921 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.171.254:42921 timed out after 15.000s (SENT)
W20250626 08:23:49.575412 2002 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.018s sys 0.045s
W20250626 08:23:49.466569 1997 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.003s sys 0.000s
W20250626 08:23:49.466691 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.004s sys 0.004s
W20250626 08:23:49.471729 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.574559 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.232065 4067 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.522338 1846 rpcz_store.cc:276] 0626 08:23:47.512256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.512376 (+ 120us) service_pool.cc:225] Handling call
0626 08:23:47.538269 (+ 25893us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:48.538683 (+1000414us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:48.538722 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:48.539260 (+ 538us) log.cc:844] Serialized 26 byte log entry
0626 08:23:48.726521 (+187261us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:48.726577 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:48.726681 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:49.461095 (+734414us) raft_consensus.cc:1683] finished
0626 08:23:49.461161 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.461564 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105277,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":12005,"thread_start_us":1186773,"threads_started":2,"wal-append.queue_time_us":552966}
I20250626 08:23:49.168992 4071 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.582599 4067 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:49.581655 1925 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:49.583827 4071 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.583942 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2010) took 1027 ms. Trace:
I20250626 08:23:49.584527 1995 rpcz_store.cc:276] 0626 08:23:48.556458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.556580 (+ 122us) service_pool.cc:225] Handling call
0626 08:23:49.583738 (+1027158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:49.585219 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.585580 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.586050 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2013) took 1028 ms. Trace:
I20250626 08:23:49.586438 1925 raft_consensus.cc:2408] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250626 08:23:49.586496 1993 rpcz_store.cc:276] 0626 08:23:48.557340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.623073 (+ 65733us) service_pool.cc:225] Handling call
0626 08:23:49.579166 (+956093us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:49.587287 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:49.588124 1926 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:48.921986 4089 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.589745 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:49.590974 2584 raft_consensus.cc:2366] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
W20250626 08:23:49.591643 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 529) took 3804 ms (client timeout 3313 ms). Trace:
I20250626 08:23:49.592474 4089 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:49.592188 2584 rpcz_store.cc:269] 0626 08:23:45.787372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.787564 (+ 192us) service_pool.cc:225] Handling call
0626 08:23:49.589274 (+3801710us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:23:49.591488 (+ 2214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4954368}
I20250626 08:23:49.249064 4132 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:49.212775 4154 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.661s user 0.017s sys 0.004s
I20250626 08:23:48.943289 4164 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.613555 4183 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.614275 4183 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:49.004858 4200 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.468s user 0.014s sys 0.003s
I20250626 08:23:49.616168 4200 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:47.940476 4206 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
W20250626 08:23:49.617853 1923 consensus_meta.cc:287] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.067s user 0.010s sys 0.001s
I20250626 08:23:49.231166 4219 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:49.243515 4244 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.619107 1923 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:49.619513 4244 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:49.621445 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 613) took 3070 ms (client timeout 1650 ms). Trace:
I20250626 08:23:49.622135 4215 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:49.622082 2583 rpcz_store.cc:269] 0626 08:23:46.550629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.586233 (+ 35604us) service_pool.cc:225] Handling call
0626 08:23:49.620943 (+3034710us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540066eed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:49.621179 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235604352}
I20250626 08:23:49.622915 4215 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:49.365127 4162 consensus_meta.cc:287] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.822s user 0.009s sys 0.002s
I20250626 08:23:49.623812 4132 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.625164 4162 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.627216 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:49.627660 4162 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:49.474252 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.004s sys 0.008s
W20250626 08:23:49.628554 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.875s user 0.005s sys 0.008s
W20250626 08:23:49.628727 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.002s sys 0.032s
W20250626 08:23:49.629146 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.072s user 0.002s sys 0.032s
W20250626 08:23:49.628953 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 929130us
I20250626 08:23:49.631140 3886 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.510943 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 432) took 2413 ms. Trace:
I20250626 08:23:49.631923 3886 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.513522 4259 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:49.633474 4259 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.634691 3886 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.516841 3668 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:49.636417 4089 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.631649 4200 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:49.523591 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.983s user 0.001s sys 0.006s
I20250626 08:23:49.637378 4089 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:49.524197 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.002s sys 0.007s
W20250626 08:23:49.525002 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.009s sys 0.001s
W20250626 08:23:49.525592 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.525774 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 430) took 3022 ms. Trace:
W20250626 08:23:49.525966 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.003s sys 0.003s
I20250626 08:23:49.545239 1920 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:23:49.546622 4157 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 3.459s user 0.011s sys 0.024s
I20250626 08:23:49.548916 2585 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:49.555934 3918 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:49.556452 1919 raft_consensus.cc:2366] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
W20250626 08:23:49.556633 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.000s sys 0.028s
I20250626 08:23:49.558535 2586 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:49.562098 1812 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.015s sys 0.041s
I20250626 08:23:49.574693 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2011) took 1017 ms. Trace:
I20250626 08:23:48.876324 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 600) took 4409 ms. Trace:
I20250626 08:23:48.804389 2581 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:49.669081 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 624) took 1115 ms. Trace:
I20250626 08:23:49.669452 2581 rpcz_store.cc:276] 0626 08:23:48.553761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.553885 (+ 124us) service_pool.cc:225] Handling call
0626 08:23:49.668876 (+1114991us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:49.578912 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.837s user 0.004s sys 0.004s
I20250626 08:23:49.602041 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:49.671365 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:49.654803 3668 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:49.472415 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.000s sys 0.004s
W20250626 08:23:49.632268 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.905s user 0.020s sys 0.041s
I20250626 08:23:49.646898 1816 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:49.647358 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1769) took 2499 ms. Trace:
I20250626 08:23:49.647821 4089 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:49.647966 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.010s sys 0.001s
W20250626 08:23:49.648356 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.004s sys 0.003s
I20250626 08:23:49.640235 1843 rpcz_store.cc:276] 0626 08:23:46.503419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.503536 (+ 117us) service_pool.cc:225] Handling call
0626 08:23:46.503877 (+ 341us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.572282 (+ 68405us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.572311 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:46.572352 (+ 41us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:46.572389 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.525165 (+2952776us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b5400211454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:49.525246 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.525547 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156832000}
W20250626 08:23:49.654145 1812 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.926s user 0.016s sys 0.041s
W20250626 08:23:49.655475 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.994s user 0.000s sys 0.029s
W20250626 08:23:49.656386 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.002s sys 0.007s
W20250626 08:23:49.671165 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 929457us
I20250626 08:23:49.660430 1926 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:49.636446 4259 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.647868 3781 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.673808 1923 rpcz_store.cc:276] 0626 08:23:47.148255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.530363 (+382108us) service_pool.cc:225] Handling call
0626 08:23:47.577227 (+ 46864us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:49.619586 (+2042359us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.647297 (+ 27711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:49.675504 3781 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.677166 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:49.666496 3982 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:49.667148 2579 rpcz_store.cc:276] 0626 08:23:44.451271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.487243 (+1035972us) service_pool.cc:225] Handling call
0626 08:23:45.487732 (+ 489us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.398202 (+910470us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:48.699381 (+2301179us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54005429d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:48.846230 (+146849us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":996480256}
I20250626 08:23:49.668233 4072 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.677867 3781 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.678067 2583 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:49.690569 4072 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.672052 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:49.603384 1921 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.936s user 0.005s sys 0.005s
I20250626 08:23:49.652572 1996 rpcz_store.cc:276] 0626 08:23:48.556790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.556917 (+ 127us) service_pool.cc:225] Handling call
0626 08:23:49.574345 (+1017428us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:49.691989 1921 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.060s user 0.006s sys 0.005s
I20250626 08:23:49.686499 4200 consensus_queue.cc:1046] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20250626 08:23:49.693599 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:49.680644 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:49.686894 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 47425.5us
I20250626 08:23:49.694919 4154 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:49.695042 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:49.695484 4154 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.695868 4258 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:49.696498 1816 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:49.696841 4255 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:49.697515 4255 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.690254 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1738) took 5771 ms. Trace:
W20250626 08:23:49.697988 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.029s user 0.021s sys 0.042s
I20250626 08:23:49.696763 4258 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:49.687063 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.865s user 0.001s sys 0.004s
W20250626 08:23:49.678463 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 950326us
I20250626 08:23:49.700075 3668 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.694294 4271 raft_consensus.cc:3053] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:49.683202 3982 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.701051 4271 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:49.682106 4219 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:49.702523 3668 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:49.692157 2584 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:49.702986 3982 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.704263 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:49.702795 4219 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.703994 4271 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.690297 3970 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:49.704385 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:49.698171 4255 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:49.705688 4089 consensus_queue.cc:1046] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:23:49.706187 4271 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:49.706374 2584 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:49.707386 4089 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:23:49.707816 4089 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.708161 4089 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:23:49.696015 4154 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:49.699033 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.670606 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:49.710608 4258 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:49.711884 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:49.631855 1924 rpcz_store.cc:276] 0626 08:23:47.096903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.106717 (+ 9814us) service_pool.cc:225] Handling call
0626 08:23:47.106980 (+ 263us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:47.540641 (+433661us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:47.540671 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:47.550120 (+ 9449us) log.cc:844] Serialized 26 byte log entry
0626 08:23:48.044174 (+494054us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:48.044233 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:48.044306 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:49.465686 (+1421380us) raft_consensus.cc:1683] finished
0626 08:23:49.465750 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.510595 (+ 44845us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4018,"raft.queue_time_us":380087,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":15156,"thread_start_us":841723,"threads_started":2,"wal-append.queue_time_us":1500977}
I20250626 08:23:49.691996 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:49.715516 2581 raft_consensus.cc:2408] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250626 08:23:49.704944 1812 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:49.705036 3668 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:49.735930 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2018) took 1124 ms. Trace:
I20250626 08:23:49.737139 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:49.736423 2002 rpcz_store.cc:276] 0626 08:23:48.611518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.611721 (+ 203us) service_pool.cc:225] Handling call
0626 08:23:48.612186 (+ 465us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:49.576413 (+964227us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:49.576519 (+ 106us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:49.577875 (+ 1356us) log.cc:844] Serialized 26 byte log entry
0626 08:23:49.664504 (+ 86629us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:49.664539 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.673028 (+ 8489us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:49.735638 (+ 62610us) raft_consensus.cc:1683] finished
0626 08:23:49.735657 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.735839 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":476464,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":1034318,"threads_started":2,"wal-append.queue_time_us":140384}
I20250626 08:23:49.738232 3759 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250626 08:23:49.739228 1967 connection.cc:232] RPC call timeout handler was delayed by -1.2089879630002542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1506s for a response
W20250626 08:23:49.739754 1967 connection.cc:232] RPC call timeout handler was delayed by -1.16308796300018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20250626 08:23:49.740134 4270 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.740898 4270 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:49.742002 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.07011e+06us
I20250626 08:23:49.744601 3918 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:49.745167 3970 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:49.749636 2584 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:49.698208 1928 rpcz_store.cc:276] 0626 08:23:43.918410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.984226 (+1065816us) service_pool.cc:225] Handling call
0626 08:23:44.984458 (+ 232us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:44.984681 (+ 223us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:44.984697 (+ 16us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:45.030844 (+ 46147us) log.cc:844] Serialized 26 byte log entry
0626 08:23:45.658824 (+627980us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:45.658873 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:45.658973 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:49.545256 (+3886283us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:49.689721 (+144465us) raft_consensus.cc:1683] finished
0626 08:23:49.689742 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.690046 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":47113,"spinlock_wait_cycles":5225856,"thread_start_us":396661,"threads_started":1,"wal-append.queue_time_us":734668}
I20250626 08:23:49.755435 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1835) took 3391 ms. Trace:
W20250626 08:23:49.755539 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 50778.7us
I20250626 08:23:49.755951 1927 rpcz_store.cc:276] 0626 08:23:46.363915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.389316 (+ 25401us) service_pool.cc:225] Handling call
0626 08:23:49.526383 (+3137067us) spinlock_profiling.cc:243] Waited 942 us on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:49.526929 (+ 546us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:49.578216 (+ 51287us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:49.578256 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:49.588146 (+ 9890us) log.cc:844] Serialized 26 byte log entry
0626 08:23:49.659753 (+ 71607us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:49.659805 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.659906 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:49.751448 (+ 91542us) raft_consensus.cc:1683] finished
0626 08:23:49.751466 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:49.755245 (+ 3779us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14130,"raft.queue_time_us":22294,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":2975,"spinlock_wait_cycles":2166016,"thread_start_us":65207,"threads_started":1,"wal-append.queue_time_us":154841}
I20250626 08:23:49.740507 1967 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:49.758188 3781 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:49.758836 3781 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:49.759330 1967 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:49.760042 3781 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:49.761032 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:49.762137 4258 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:49.762499 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:49.763136 4258 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 08:23:49.763684 3970 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:49.765574 2578 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.578s user 0.016s sys 0.004s
I20250626 08:23:49.766053 2578 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:49.764092 4258 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:49.729372 3668 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.774160 4219 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:49.775216 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:49.776258 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.777967 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1962) took 5271 ms. Trace:
I20250626 08:23:49.778264 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.778983 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:49.789736 4212 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:49.790416 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:49.747080 4270 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.791522 1998 rpcz_store.cc:276] 0626 08:23:44.506725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.625223 (+1118498us) service_pool.cc:225] Handling call
0626 08:23:45.625528 (+ 305us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:45.648511 (+ 22983us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.777397 (+4128886us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b540039d854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:49.777741 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44089216}
I20250626 08:23:49.923429 1997 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:49.923885 3982 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:49.806381 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:49.852166 1923 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:49.813547 1927 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:49.749267 3918 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.929759 4270 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:49.794597 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 114301us
W20250626 08:23:49.932896 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:49.929963 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 169049us
I20250626 08:23:49.791181 3668 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:49.955395 1925 raft_consensus.cc:2391] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:49.932340 4270 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:49.956455 1965 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:49.935262 3918 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:49.958251 4270 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:49.958856 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.959121 4062 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:49.959509 4062 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:49.959836 4062 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:49.963521 3978 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:49.958391 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.918623 4269 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:49.964190 3978 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
W20250626 08:23:49.971750 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:49.972263 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 197224us
I20250626 08:23:49.917894 4271 consensus_queue.cc:1046] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.051s
I20250626 08:23:49.978569 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:49.980796 4256 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 08:23:49.981077 3978 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:49.981573 4256 raft_consensus.cc:695] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:49.978861 4271 raft_consensus.cc:3053] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:49.985263 4271 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:49.990445 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:49.992275 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:49.995370 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:49.995901 1919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 427) took 2901 ms (client timeout 1626 ms). Trace:
W20250626 08:23:49.996469 1919 rpcz_store.cc:269] 0626 08:23:47.094138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.095771 (+ 1633us) service_pool.cc:225] Handling call
0626 08:23:49.995554 (+2899783us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b54003683d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:49.995784 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5321050368}
W20250626 08:23:50.002841 4277 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250626 08:23:50.005074 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.007776 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:50.008602 3978 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250626 08:23:50.009363 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.009500 4256 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:50.015656 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 50498.2us
W20250626 08:23:49.989181 2551 connection.cc:232] RPC call timeout handler was delayed by -2.5489233909997893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20250626 08:23:50.027377 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 325558us
I20250626 08:23:50.032045 4242 consensus_queue.cc:1046] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:23:50.029609 1816 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:50.036039 1890 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:50.036379 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 48540.4us
I20250626 08:23:50.037302 1816 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:50.037722 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.036983 4255 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:50.046224 2551 connection.cc:232] RPC call timeout handler was delayed by -2.36712339099995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
I20250626 08:23:50.047817 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.053510 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.062639 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.063190 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.064863 2551 connection.cc:232] RPC call timeout handler was delayed by -1.3306569649998892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20250626 08:23:50.066591 2551 connection.cc:232] RPC call timeout handler was delayed by -1.228056965000178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20379999999999998s for a response
W20250626 08:23:50.066938 2551 connection.cc:232] RPC call timeout handler was delayed by -1.0923694129996875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20250626 08:23:50.073338 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.039006 4154 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.044355 4269 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.071470 2583 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:50.077668 2584 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:50.097091 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:50.097513 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.068851 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.077409 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.100739 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.102186 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.071609 4270 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:50.072630 4132 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:50.103539 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.105898 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.094981 3978 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:50.108004 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 1969) took 5518 ms. Trace:
I20250626 08:23:50.110056 4211 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:50.111348 4195 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.111434 4269 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.076872 3976 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:50.121017 4188 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.128233 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.150058 3952 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:50.150244 3976 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:50.150529 4258 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:50.150774 3976 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.128944 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.105376 4270 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:50.151572 4270 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.151865 1850 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:50.153241 1924 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.115625 4255 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.116110 4216 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:50.153978 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.154072 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.155146 1843 raft_consensus.cc:2408] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:23:50.116616 4117 raft_consensus.cc:3053] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:50.155512 4117 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:50.156296 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 110273us
I20250626 08:23:50.156700 4117 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.157181 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.116686 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.136718 3978 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:50.140494 4089 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.159497 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.160285 4089 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.160960 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.117650 4214 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.162110 1928 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:50.120177 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.164686 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.165269 1847 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.166265 4211 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:50.102310 4200 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:50.121408 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44329.6us
I20250626 08:23:50.128285 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.168249 4200 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.168998 1920 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:23:50.170040 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.112140 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:50.172034 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 60080.5us
I20250626 08:23:50.172554 4259 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.172878 4117 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 08:23:50.173517 4159 consensus_queue.cc:1046] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250626 08:23:50.126567 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 66809.5us
I20250626 08:23:50.175482 1923 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:50.118413 4285 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:50.157397 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.143236 4154 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.151401 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.158320 3978 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.190234 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.150791 3952 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.136790 2000 rpcz_store.cc:276] 0626 08:23:44.589531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.629259 (+1039728us) service_pool.cc:225] Handling call
0626 08:23:45.637402 (+ 8143us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:46.418264 (+780862us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:50.107469 (+3689205us) spinlock_profiling.cc:243] Waited 776 ms on lock 0x7b540018ea54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:50.107800 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1784973696}
I20250626 08:23:50.193013 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.131812 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.195901 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.196599 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.183183 1847 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:50.190232 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.198908 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.200299 2581 raft_consensus.cc:2366] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
I20250626 08:23:50.213757 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.230919 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.232364 2550 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:50.236290 2551 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:50.239209 1928 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:23:50.239951 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:50.224815 4193 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 2.864s user 0.034s sys 0.063s
I20250626 08:23:50.247395 1848 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.196494 3918 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.253134 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.195825 4211 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:50.243257 4193 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 3.073s user 0.034s sys 0.064s
W20250626 08:23:50.258445 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 101357us
I20250626 08:23:50.234458 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.207947 4132 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.239516 4214 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.264315 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.265059 1965 leader_election.cc:400] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 2 election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3.
W20250626 08:23:50.297040 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 74824.5us
I20250626 08:23:50.297888 2550 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:50.312211 1923 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:50.314198 1850 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250626 08:23:50.317469 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.314616 1850 pending_rounds.cc:85] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting all ops after (but not including) 3
I20250626 08:23:50.236013 4285 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.317482 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:50.322615 1928 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:23:50.324206 1843 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:50.281783 4200 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.301743 4159 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.320376 1850 pending_rounds.cc:107] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250626 08:23:50.328167 1850 raft_consensus.cc:2887] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:50.270824 4193 raft_consensus.cc:695] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:50.333318 1927 raft_consensus.cc:2408] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250626 08:23:50.337245 1923 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:50.338625 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 96523.6us
I20250626 08:23:50.341531 1845 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:23:50.342240 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.343091 4159 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:50.343533 4159 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:50.343917 4159 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.344229 4244 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.352154 2000 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:23:50.352815 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.354053 2579 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:23:50.354653 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 437) took 1797 ms. Trace:
I20250626 08:23:50.374354 1921 rpcz_store.cc:276] 0626 08:23:48.557294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.563991 (+ 6697us) service_pool.cc:225] Handling call
0626 08:23:48.564263 (+ 272us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:49.692871 (+1128608us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:49.692923 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:49.693594 (+ 671us) log.cc:844] Serialized 26 byte log entry
0626 08:23:49.826068 (+132474us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:49.826122 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:49.892419 (+ 66297us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:50.354394 (+461975us) raft_consensus.cc:1683] finished
0626 08:23:50.354415 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:50.354577 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":280896,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":40778,"thread_start_us":1192373,"threads_started":2,"wal-append.queue_time_us":657611}
I20250626 08:23:50.375679 4083 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:50.376333 4083 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:50.376350 4072 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.379748 4285 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:50.381476 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 166641us
I20250626 08:23:50.355429 4270 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.358392 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 617) took 3391 ms. Trace:
W20250626 08:23:50.359391 2587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 611) took 3809 ms (client timeout 3292 ms). Trace:
I20250626 08:23:50.359719 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:50.360546 1918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1768) took 3212 ms (client timeout 2738 ms). Trace:
I20250626 08:23:50.360926 3918 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.361595 2582 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:50.344776 3982 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:50.387874 2585 rpcz_store.cc:276] 0626 08:23:46.966890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.982481 (+ 15591us) service_pool.cc:225] Handling call
0626 08:23:46.982824 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:49.662610 (+2679786us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:50.358046 (+695436us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b54005442d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:50.358293 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5895716608}
W20250626 08:23:50.388600 2587 rpcz_store.cc:269] 0626 08:23:46.550266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.767098 (+216832us) service_pool.cc:225] Handling call
0626 08:23:50.358965 (+3591867us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400500354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:50.359186 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4555702144}
W20250626 08:23:50.364097 4226 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.424s user 0.025s sys 0.018s
I20250626 08:23:50.362452 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.411093 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.411985 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.412434 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:49.776891 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:50.413126 1919 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:50.364795 3851 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:50.414974 4270 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.415877 1843 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.417825 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.424417 3715 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.427376 3973 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:50.365515 4231 log.cc:927] Time spent T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 2.424s user 0.034s sys 0.060s
I20250626 08:23:50.366864 1845 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:50.368346 4253 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:23:50.388831 1918 rpcz_store.cc:269] 0626 08:23:47.147783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.323133 (+175350us) service_pool.cc:225] Handling call
0626 08:23:50.360206 (+3037073us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54001f8e54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:50.360355 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":692881280}
I20250626 08:23:50.389662 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.359407 3952 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.363445 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.394774 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.395954 4289 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250626 08:23:50.395859 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.397964 4164 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:50.394796 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 113729us
I20250626 08:23:50.398792 4132 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:50.401321 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.402833 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.406574 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.407984 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.408888 3918 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:50.409215 4062 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.425001 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.425421 1998 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.415900 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1675) took 3090 ms. Trace:
I20250626 08:23:50.426450 4189 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:50.389448 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 266798us
I20250626 08:23:50.431841 2584 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:23:50.428154 1893 connection.cc:232] RPC call timeout handler was delayed by -6.4607572110003275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20250626 08:23:50.421738 4259 raft_consensus.cc:3053] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:50.439249 1924 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:50.446978 2585 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:50.448161 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.335211 3817 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.216s
W20250626 08:23:50.467034 1893 connection.cc:232] RPC call timeout handler was delayed by -6.0728572110001551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
W20250626 08:23:50.479125 1893 connection.cc:232] RPC call timeout handler was delayed by -5.4277352720000636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20250626 08:23:50.479482 1893 connection.cc:232] RPC call timeout handler was delayed by -5.2108352720001676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
W20250626 08:23:50.479836 1893 connection.cc:232] RPC call timeout handler was delayed by -4.907535272000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2377s for a response
W20250626 08:23:50.480118 1893 connection.cc:232] RPC call timeout handler was delayed by -4.3099352720000752s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30410000000000004s for a response
I20250626 08:23:50.377350 4083 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:50.471252 2587 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:50.468304 1843 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.467759 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.472368 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:50.475270 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.475517 2584 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:50.476815 2585 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:50.477231 1746 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46332 (request call id 189) took 16186 ms (client timeout 15000 ms). Trace:
W20250626 08:23:50.490869 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 58578.8us
I20250626 08:23:50.453677 4253 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:50.490818 2586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 607) took 4649 ms (client timeout 3310 ms). Trace:
W20250626 08:23:50.496595 2586 rpcz_store.cc:269] 0626 08:23:45.836716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:45.837127 (+ 411us) service_pool.cc:225] Handling call
0626 08:23:50.478259 (+4641132us) spinlock_profiling.cc:243] Waited 3.71 s on lock 0x7b540044d754. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:50.486028 (+ 7769us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8533715840}
I20250626 08:23:50.479537 4270 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:50.499235 4270 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.507588 4271 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.508801 4271 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:50.490196 1746 rpcz_store.cc:269] 0626 08:23:34.290933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:34.295577 (+ 4644us) service_pool.cc:225] Handling call
0626 08:23:35.527249 (+1231672us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54001df954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003a70 00007f4eb9087b1a 00007f4ebed9afac 00007f4ebee937ee 00007f4ebd3dd624 00007f4ebd3dd5b2 00007f4ebd3dd4e1 00007f4ebd3dd45c 00007f4ebd3dc712 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:50.476937 (+14949688us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:46.954057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:47.121497 (+167440us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:47.121536 (+ 39us) write_op.cc:435] Acquired schema lock
0626 08:23:47.121557 (+ 21us) tablet.cc:660] Decoding operations
0626 08:23:47.122550 (+ 993us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:47.122635 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:47.122668 (+ 33us) tablet.cc:683] Acquiring locks for 6 operations
0626 08:23:47.123274 (+ 606us) tablet.cc:699] Row locks acquired
0626 08:23:47.123296 (+ 22us) write_op.cc:260] PREPARE: finished
0626 08:23:47.123455 (+ 159us) write_op.cc:270] Start()
0626 08:23:47.123580 (+ 125us) write_op.cc:276] Timestamp: P: 1750926227123401 usec, L: 0
0626 08:23:47.123602 (+ 22us) op_driver.cc:348] REPLICATION: starting
0626 08:23:47.234357 (+110755us) log.cc:844] Serialized 2452 byte log entry
0626 08:23:49.251573 (+2017216us) op_driver.cc:464] REPLICATION: finished
0626 08:23:49.563161 (+311588us) write_op.cc:301] APPLY: starting
0626 08:23:49.563385 (+ 224us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:49.566580 (+ 3195us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:49.566800 (+ 220us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:49.581984 (+ 15184us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:49.582419 (+ 435us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0626 08:23:49.582515 (+ 96us) write_op.cc:312] APPLY: finished
0626 08:23:49.583024 (+ 509us) log.cc:844] Serialized 53 byte log entry
0626 08:23:49.583383 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:49.583670 (+ 287us) write_op.cc:454] Released schema lock
0626 08:23:49.583737 (+ 67us) write_op.cc:341] FINISH: Updating metrics
0626 08:23:49.602797 (+ 19060us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:23:49.602838 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1286410,"reported_tablets":8,"spinlock_wait_cycles":876302208,"child_traces":[["op",{"apply.queue_time_us":311042,"apply.run_cpu_time_us":4740,"apply.run_wall_time_us":39680,"num_ops":6,"prepare.queue_time_us":579039,"prepare.run_cpu_time_us":50861,"prepare.run_wall_time_us":1149613,"raft.queue_time_us":380636,"raft.run_cpu_time_us":2617,"raft.run_wall_time_us":121704,"replication_time_us":2127785,"thread_start_us":1265781,"threads_started":4,"wal-append.queue_time_us":445080}]]}
I20250626 08:23:50.501675 4270 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.510082 4189 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.510673 3566 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:23:50.512893 4189 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.514245 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.515261 1996 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:50.515403 1967 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.497716 1920 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:50.516907 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.473233 4259 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:50.517897 4014 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.518110 4038 raft_consensus.cc:3053] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:50.518215 3994 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.518620 4038 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:50.501286 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.519855 2580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 605) took 4255 ms (client timeout 2000 ms). Trace:
I20250626 08:23:50.519668 4038 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.520108 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:50.520330 2580 rpcz_store.cc:269] 0626 08:23:46.264647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.283826 (+ 19179us) service_pool.cc:225] Handling call
0626 08:23:50.488863 (+4205037us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:23:50.519776 (+ 30913us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55929600}
I20250626 08:23:50.514230 4213 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:50.520813 4038 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:50.502168 2579 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:50.521957 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.546496 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
W20250626 08:23:50.547150 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 60594.6us
I20250626 08:23:50.544485 1918 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:50.544389 1968 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.533834 1998 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:50.544440 4213 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:50.558144 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.559096 1996 raft_consensus.cc:2391] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:50.510109 4307 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:50.560423 4307 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.562726 4307 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.562438 4253 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.426487 4193 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:50.566066 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.500643 3982 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20250626 08:23:50.572060 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:50.573467 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 79908.6us
I20250626 08:23:50.468370 1849 rpcz_store.cc:276] 0626 08:23:47.325365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.325899 (+ 534us) service_pool.cc:225] Handling call
0626 08:23:47.426253 (+100354us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:47.426631 (+ 378us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:47.426659 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:47.427117 (+ 458us) log.cc:844] Serialized 26 byte log entry
0626 08:23:47.664580 (+237463us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:47.664639 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:47.664735 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:50.363261 (+2698526us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:50.415015 (+ 51754us) raft_consensus.cc:1683] finished
0626 08:23:50.415039 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:50.415741 (+ 702us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26270,"raft.queue_time_us":19031,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":40842,"spinlock_wait_cycles":10767744,"thread_start_us":237126,"threads_started":1,"wal-append.queue_time_us":483357}
I20250626 08:23:50.549414 3994 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.574033 4167 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:50.487445 3973 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:50.591518 2551 leader_election.cc:336] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.800s (SENT)
I20250626 08:23:50.597297 2551 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.592674 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.589748 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.574486 4164 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:50.595252 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.590184 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:23:50.606091 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 59700.5us
I20250626 08:23:50.574240 4062 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.606516 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.602538 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:50.608759 1845 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:23:50.609061 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.613811 2586 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:50.614856 4195 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.615833 3973 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:50.616546 4161 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.616719 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.617542 1926 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.619184 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:50.614002 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.619300 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 434) took 3421 ms. Trace:
W20250626 08:23:50.626670 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 106698us
I20250626 08:23:50.630553 1996 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:23:50.630697 1927 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:50.631417 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:50.632532 4271 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:50.632639 1814 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:50.632989 4271 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:50.633453 4271 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:50.635069 4276 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.678s user 0.002s sys 0.004s
I20250626 08:23:50.636605 4189 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:50.637112 4189 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:50.637524 4189 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:50.643291 4119 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:50.644325 4119 consensus_queue.cc:1230] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250626 08:23:50.646570 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.621722 1841 rpcz_store.cc:276] 0626 08:23:47.197680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.198482 (+ 802us) service_pool.cc:225] Handling call
0626 08:23:47.341761 (+143279us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:49.546730 (+2204969us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:50.619046 (+1072316us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400255dd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:50.619250 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4870432000}
I20250626 08:23:50.609722 4211 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:50.609591 4223 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.605406 4164 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:50.649112 1812 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.610117 2580 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:23:50.653362 1968 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.655057 1843 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:50.645251 3994 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:50.658345 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.658654 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.626375 4259 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.661872 1890 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:50.666568 1849 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.666174 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.667840 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.610550 3976 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.652841 4164 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:50.668972 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:50.674197 1844 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 436) took 3474 ms (client timeout 1800 ms). Trace:
W20250626 08:23:50.674628 1844 rpcz_store.cc:269] 0626 08:23:47.199418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:47.226697 (+ 27279us) service_pool.cc:225] Handling call
0626 08:23:50.656413 (+3429716us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54000348d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:50.674003 (+ 17590us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1004516736}
I20250626 08:23:50.674926 4183 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:50.675858 4291 log.cc:927] Time spent T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.411s user 0.001s sys 0.000s
I20250626 08:23:50.675626 3982 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:50.676100 1844 raft_consensus.cc:3053] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:50.676520 1844 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:50.676939 3851 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:50.678853 4011 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.680019 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.665891 3994 raft_consensus.cc:695] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:50.682936 3994 consensus_queue.cc:237] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.603657 4167 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.687261 1968 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.688503 4167 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.688675 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.688720 3781 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.689126 4167 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.689497 4183 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.690543 1998 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
W20250626 08:23:50.691051 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.673866 3978 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.677284 1844 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.668445 4117 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:50.665684 4259 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.701395 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:50.692232 3668 consensus_meta.cc:287] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.708s user 0.006s sys 0.006s
W20250626 08:23:50.705509 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 75922.2us
I20250626 08:23:50.706346 1812 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:50.708583 1849 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:50.709342 1850 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.717281 1925 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:50.693193 4195 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:50.716739 4188 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250626 08:23:50.720273 1843 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:50.720418 1845 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:50.696907 4167 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:50.740582 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.696494 4117 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:50.752141 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 45920.8us
I20250626 08:23:50.743211 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.754073 4119 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:50.754645 4119 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:50.755687 4119 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.758054 1844 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:50.758391 2586 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:50.761879 1926 raft_consensus.cc:2366] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:50.768309 1965 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:50.768226 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 51160.4us
I20250626 08:23:50.772346 1967 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.776101 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.776688 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 623) took 2223 ms. Trace:
I20250626 08:23:50.717005 3851 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20250626 08:23:50.777334 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.781250 1843 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:50.777303 4189 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:50.782119 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.751052 4167 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:50.789425 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.778095 4269 consensus_queue.cc:1046] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20250626 08:23:50.777088 2582 rpcz_store.cc:276] 0626 08:23:48.553071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:48.553179 (+ 108us) service_pool.cc:225] Handling call
0626 08:23:50.361004 (+1807825us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:50.390950 (+ 29946us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:50.776519 (+385569us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400500350. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:50.776634 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4156842368}
W20250626 08:23:50.787875 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.799957 1844 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20250626 08:23:50.800559 1844 pending_rounds.cc:85] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting all ops after (but not including) 5
I20250626 08:23:50.800941 1844 pending_rounds.cc:107] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20250626 08:23:50.801327 1844 raft_consensus.cc:2887] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:50.717463 3668 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.804735 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.807773 1926 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.740599 4062 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20250626 08:23:50.808173 1814 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.810741 1967 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.811007 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.811815 2583 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:23:50.762445 4117 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.815789 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.823020 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.823240 1842 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
I20250626 08:23:50.817404 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:50.825169 2000 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250626 08:23:50.826263 1919 raft_consensus.cc:1238] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:50.743300 4223 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250626 08:23:50.741629 4211 consensus_queue.cc:1046] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20250626 08:23:50.793787 4188 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:50.828517 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.828722 2550 leader_election.cc:400] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
I20250626 08:23:50.839937 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:50.831228 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.795745 4167 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:50.802070 3978 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.788998 3781 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250626 08:23:50.843327 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.845495 1893 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:50.838766 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.853415 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.836570 1842 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:50.848659 1814 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.852154 2581 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:50.847440 1968 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:50.807379 4307 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:50.859802 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:50.860775 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 50129.4us
I20250626 08:23:50.861768 1967 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.851362 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.865516 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.865765 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:50.866945 2586 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:50.865576 1995 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:50.869189 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.869897 1849 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:23:50.836280 4188 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:50.842933 4078 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250626 08:23:50.843014 4014 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:50.892999 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.846705 4270 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:23:50.893996 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.890002 2551 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.890619 1814 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:50.875588 1847 raft_consensus.cc:2391] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:50.889096 3566 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250626 08:23:50.901052 1844 raft_consensus.cc:2408] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:23:50.901139 4264 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:50.902158 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.902782 1844 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:50.904824 1995 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:50.905983 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.907114 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.829324 3851 consensus_queue.cc:1046] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20250626 08:23:50.908154 3851 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:50.908582 3851 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:50.909555 3851 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.910955 3851 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:50.912006 3936 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:50.873514 4038 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:50.914285 4308 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:50.914806 4308 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:50.915170 4308 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:50.916898 3976 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.916939 1919 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:50.872382 4214 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:50.917807 4214 raft_consensus.cc:695] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:50.918888 3668 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.918753 4214 consensus_queue.cc:237] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:50.919647 4156 raft_consensus.cc:1062] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:23:50.919445 4204 consensus_queue.cc:1046] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:23:50.920552 4204 raft_consensus.cc:3053] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:50.920879 4204 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:50.922531 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.923681 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.921835 4204 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.925283 4204 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:50.926401 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.873891 3759 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:50.922531 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 77139.8us
I20250626 08:23:50.915061 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.930469 1843 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:50.908339 4290 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20250626 08:23:50.891127 4188 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:50.908165 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:50.929463 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.868877 4253 consensus_queue.cc:1046] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
W20250626 08:23:50.934036 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 82330us
I20250626 08:23:50.944416 1845 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:50.944862 2551 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:50.900985 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:50.947114 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:50.947800 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:50.948424 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57890.2us
I20250626 08:23:50.904844 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:50.926024 4203 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:50.956523 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.947608 1995 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:23:50.957630 1849 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:50.958370 4253 raft_consensus.cc:3053] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:50.959167 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:50.974570 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:50.961321 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.961944 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.959636 1845 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:50.974609 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.973130 1844 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:23:50.907106 3918 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:50.945291 3952 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:50.980674 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:50.987807 1844 raft_consensus.cc:1238] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:50.962620 4269 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:50.991776 1919 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:50.988243 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:50.994742 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:50.995251 1847 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:23:50.995960 1923 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:50.996376 1849 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:23:50.958465 4189 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.963279 4203 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:51.024150 4316 log.cc:927] Time spent T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250626 08:23:51.104426 2581 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:51.023665 1843 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:23:51.000134 2002 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.031880 4215 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:50.973922 4195 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:51.035619 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:51.060992 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:51.058835 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:50.993209 4251 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:51.034670 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.118670 1844 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:51.033565 4164 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.036839 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:51.026743 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.027855 4167 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.104802 4203 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:23:51.050138 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 52086us
I20250626 08:23:51.093499 1845 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:51.069867 1923 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:23:51.072050 2551 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:51.101292 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.099068 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.068696 4205 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 3.641s user 0.049s sys 0.064s
I20250626 08:23:51.114351 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.116703 1849 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:51.043839 3952 raft_consensus.cc:695] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.204284 3952 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:51.205006 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 106048us
I20250626 08:23:51.206254 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:51.115257 4195 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:51.207026 4195 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.208112 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:51.209944 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2008) took 4254 ms. Trace:
I20250626 08:23:51.210354 1999 rpcz_store.cc:276] 0626 08:23:46.955328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:46.955443 (+ 115us) service_pool.cc:225] Handling call
0626 08:23:46.957466 (+ 2023us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:46.975027 (+ 17561us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:46.975079 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:46.986319 (+ 11240us) log.cc:844] Serialized 26 byte log entry
0626 08:23:47.109772 (+123453us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:47.589410 (+479638us) log.cc:844] Serialized 14 byte log entry
0626 08:23:47.628967 (+ 39557us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:47.629163 (+ 196us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:51.049191 (+3420028us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:51.209467 (+160276us) raft_consensus.cc:1683] finished
0626 08:23:51.209568 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:51.209738 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111139,"prepare.queue_time_us":597104,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":144350,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":25087,"spinlock_wait_cycles":6249856,"thread_start_us":473918,"threads_started":2,"wal-append.queue_time_us":154837}
I20250626 08:23:51.212378 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 627) took 1179 ms. Trace:
I20250626 08:23:51.212880 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:51.215124 4167 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.215369 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:51.216569 4183 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.527s user 0.003s sys 0.006s
I20250626 08:23:51.115316 4189 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20250626 08:23:51.216742 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:51.220391 4259 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.516s user 0.009s sys 0.000s
I20250626 08:23:51.187572 4251 raft_consensus.cc:695] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:51.222922 4251 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.213846 1999 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:51.224915 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:51.225504 1999 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.226439 1841 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:51.228027 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 549) took 1103 ms. Trace:
W20250626 08:23:51.146016 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:51.228502 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.228400 2583 rpcz_store.cc:276] 0626 08:23:50.124929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.199850 (+ 74921us) service_pool.cc:225] Handling call
0626 08:23:51.227868 (+1028018us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54006149d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:51.227971 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1317775104}
I20250626 08:23:51.232070 2579 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:51.234493 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1862) took 1060 ms. Trace:
I20250626 08:23:51.234876 1920 rpcz_store.cc:276] 0626 08:23:50.174195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.176602 (+ 2407us) service_pool.cc:225] Handling call
0626 08:23:51.234268 (+1057666us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b5400367ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:51.234396 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":698511232}
I20250626 08:23:51.235103 4213 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:51.236280 4213 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.238127 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1857) took 1172 ms. Trace:
I20250626 08:23:51.238540 1925 rpcz_store.cc:276] 0626 08:23:50.070609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.070787 (+ 178us) service_pool.cc:225] Handling call
0626 08:23:51.237862 (+1167075us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400260f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:51.238061 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1394459904}
I20250626 08:23:51.238945 4189 consensus_queue.cc:1046] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:23:51.239969 3973 raft_consensus.cc:3053] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:51.240327 3973 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:51.240836 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:51.241828 4189 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:51.241212 3973 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.242251 4189 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:51.242762 3973 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.244565 3982 raft_consensus.cc:3053] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:51.244976 3982 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.248750 1893 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:51.249608 4215 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:51.182608 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:51.189539 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.250949 2001 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:23:51.111126 4290 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:51.252723 4290 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:51.254390 4271 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.620s user 0.009s sys 0.000s
I20250626 08:23:51.254833 4271 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.255328 4290 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.197710 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:51.061846 4269 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.216897 4164 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.242821 4189 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.256830 4271 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:51.257876 4159 consensus_meta.cc:287] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.913s user 0.011s sys 0.002s
I20250626 08:23:51.257984 4269 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:51.258430 4159 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.259757 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:51.193598 1846 raft_consensus.cc:2391] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:51.260432 4159 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:51.262184 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:51.260442 1926 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:50.947260 4156 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:50.973865 4253 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:51.250840 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:51.279758 1999 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:51.284816 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 76785us
I20250626 08:23:51.212838 2578 rpcz_store.cc:276] 0626 08:23:50.033243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.211271 (+1178028us) service_pool.cc:225] Handling call
0626 08:23:51.211680 (+ 409us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.211927 (+ 247us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.211969 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:51.212031 (+ 62us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:23:51.212078 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.212154 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:51.212291 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:51.221031 1742 catalog_manager.cc:5582] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 2 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:23:51.094506 4117 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.282123 1995 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
W20250626 08:23:51.289728 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 217756us
I20250626 08:23:51.305312 2551 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:51.294672 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.216991 4183 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:23:51.307870 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 57100.8us
I20250626 08:23:51.299926 1816 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:51.306545 1926 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.309693 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:51.311440 1850 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
I20250626 08:23:51.307785 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.312461 1924 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:51.307337 2001 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:51.310959 1993 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:51.313722 1918 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:23:51.314559 2001 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:51.315941 4117 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.316735 2001 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250626 08:23:51.319504 1814 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.320129 4308 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:51.320725 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.321048 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:51.322173 4109 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250626 08:23:51.290874 4203 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.322664 4109 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:51.323022 4109 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.323334 4109 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.323530 4056 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:51.323942 4203 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:51.324676 1849 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250626 08:23:51.326104 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.326644 1919 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:51.327664 4253 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.327782 4189 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.328649 4215 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:23:51.328784 3978 consensus_queue.cc:1046] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:51.329201 4215 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:51.329996 4189 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.329902 3978 consensus_queue.cc:1046] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:51.330158 4215 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.331135 4253 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.331518 4189 raft_consensus.cc:3053] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 08:23:51.331913 4189 raft_consensus.cc:738] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:51.332732 4189 consensus_queue.cc:260] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.289543 3668 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:51.291654 3781 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:51.334445 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.325745 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.307224 4204 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:51.336637 1814 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.337270 1965 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:51.347235 1842 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:51.336134 1812 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:51.365330 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.365737 3973 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:23:51.388607 3973 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:51.389057 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.004s sys 0.035s
W20250626 08:23:51.389374 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.004s sys 0.035s
I20250626 08:23:51.389604 3973 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.391017 3973 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:23:51.366827 3813 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:51.394658 3813 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.395099 3813 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.367755 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:51.367830 1923 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250626 08:23:51.400873 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:51.401011 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.401726 1923 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:51.368178 2001 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.368986 2587 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []
I20250626 08:23:51.369318 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:51.373267 1847 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:23:51.404693 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 68125.7us
I20250626 08:23:51.373320 1998 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:51.406798 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 69624.8us
W20250626 08:23:51.389631 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 534164us
I20250626 08:23:51.381443 1926 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:51.372268 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:51.376461 1742 catalog_manager.cc:5582] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 2 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:51.415128 1924 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
W20250626 08:23:51.415671 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 79611.6us
I20250626 08:23:51.405615 1814 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.429898 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.430920 1926 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.432107 1924 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.333827 4189 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:51.427960 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:51.403734 4193 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.014s sys 0.044s
I20250626 08:23:51.433382 1742 catalog_manager.cc:5582] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 3 to 4, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } }
I20250626 08:23:51.444705 1742 catalog_manager.cc:5582] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:51.431557 1918 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:51.446926 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.447199 1968 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:51.386621 4307 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:23:51.437381 1921 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.023s sys 0.030s
I20250626 08:23:51.306946 4262 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.328s
I20250626 08:23:51.368985 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:51.449446 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.371915 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:51.366564 4117 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:51.376091 4271 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:51.373050 4308 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:51.453552 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1779) took 1124 ms. Trace:
W20250626 08:23:51.453850 1921 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.023s sys 0.031s
I20250626 08:23:51.434967 4038 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:51.461555 1927 rpcz_store.cc:276] 0626 08:23:50.329575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.363130 (+ 33555us) service_pool.cc:225] Handling call
0626 08:23:51.453380 (+1090250us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54002d05d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:51.453496 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":441690368}
I20250626 08:23:51.414034 4119 raft_consensus.cc:2762] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250626 08:23:51.430931 4062 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250626 08:23:51.443671 4193 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.014s sys 0.045s
W20250626 08:23:51.462894 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 163078us
I20250626 08:23:51.464180 1816 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:51.464499 1814 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.465034 3566 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:51.465417 3566 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.465745 3566 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:51.465746 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:51.460175 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:51.444566 4214 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.019s sys 0.027s
I20250626 08:23:51.469277 1742 catalog_manager.cc:5582] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 3 to 4, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 4 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:51.459393 1918 raft_consensus.cc:1238] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:51.460142 2001 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:51.472561 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:51.493422 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.453089 3886 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:51.502359 1927 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:23:51.502771 1814 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.503566 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.452659 4213 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:51.503604 1814 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.505018 2583 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:51.375155 4195 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.462568 4183 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:23:51.544430 2585 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250626 08:23:51.492897 4214 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.023s sys 0.027s
I20250626 08:23:51.494458 3844 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20250626 08:23:51.495180 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:51.509311 2587 raft_consensus.cc:1273] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:23:51.510388 1742 catalog_manager.cc:5582] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194). New cstate: current_term: 2 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:23:51.513554 1926 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:51.496479 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:51.534058 3791 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
W20250626 08:23:51.534858 4317 log.cc:927] Time spent T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20250626 08:23:51.463171 4117 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:51.495139 4056 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:51.543102 3886 raft_consensus.cc:695] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:51.496491 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.462217 4038 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.469133 4062 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.553337 1918 raft_consensus.cc:1238] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.473194 4223 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:51.565302 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:51.569849 1842 raft_consensus.cc:2391] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:51.564785 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:51.560416 1742 catalog_manager.cc:5582] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 2 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:23:51.564719 4038 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.547416 4269 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:51.544020 3781 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:23:51.544612 4211 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.008s sys 0.016s
I20250626 08:23:51.568300 1927 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:23:51.546890 4183 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:51.565647 4188 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.619s user 0.011s sys 0.000s
I20250626 08:23:51.573858 3844 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:51.574247 3781 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:51.574695 4269 raft_consensus.cc:695] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.575510 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.596647 1842 raft_consensus.cc:2391] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:51.596575 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1701) took 1337 ms. Trace:
I20250626 08:23:51.578128 4056 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:51.509790 4213 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:51.568521 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 72121us
I20250626 08:23:51.578861 3973 consensus_queue.cc:1046] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250626 08:23:51.553457 1814 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.583046 3566 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.592689 1924 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.564886 4062 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:51.562110 4117 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.513988 3668 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20250626 08:23:51.595260 1742 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 3 to 5, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 5 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:23:51.596091 3781 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:51.569413 3791 raft_consensus.cc:695] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.547106 4164 consensus_queue.cc:1046] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
W20250626 08:23:51.586398 4211 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.009s sys 0.016s
I20250626 08:23:51.601840 1816 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:23:51.610093 1923 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:23:51.591761 4183 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:23:51.618245 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:51.618902 1842 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:51.622473 4322 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20250626 08:23:51.624032 4269 consensus_queue.cc:237] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:51.626647 4328 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20250626 08:23:51.601521 4213 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:51.583961 4223 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.632799 4223 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.609174 4203 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:51.634431 4203 raft_consensus.cc:695] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.634492 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:51.635370 4203 consensus_queue.cc:237] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.597173 1848 rpcz_store.cc:276] 0626 08:23:50.258968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.265444 (+ 6476us) service_pool.cc:225] Handling call
0626 08:23:50.297785 (+ 32341us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:50.380976 (+ 83191us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:50.427558 (+ 46582us) log.cc:844] Serialized 14 byte log entry
0626 08:23:50.667001 (+239443us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:50.667073 (+ 72us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:50.667135 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.596351 (+929216us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b5400210f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:51.596400 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:51.596520 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12642,"prepare.queue_time_us":41520,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":34119,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"spinlock_wait_cycles":168617728,"thread_start_us":226172,"threads_started":1,"wal-append.queue_time_us":396148}
I20250626 08:23:51.637647 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:51.562279 4119 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
W20250626 08:23:51.600239 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46885.1us
I20250626 08:23:51.612823 3566 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:51.613704 3886 consensus_queue.cc:237] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.614393 4056 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:51.614089 1742 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:23:51.614851 3844 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:51.614961 3791 consensus_queue.cc:237] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.615693 4164 raft_consensus.cc:3053] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:23:51.678340 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:51.682649 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:51.612156 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.684135 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 72064.2us
I20250626 08:23:51.684911 1816 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:51.649524 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:51.685806 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.685949 1926 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:51.645500 1842 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
W20250626 08:23:51.655424 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.022s sys 0.046s
W20250626 08:23:51.690514 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.022s sys 0.046s
W20250626 08:23:51.691190 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 829462us
I20250626 08:23:51.692000 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.670125 2000 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.010s sys 0.020s
I20250626 08:23:51.681545 3566 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:23:51.682585 4164 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.677309 1742 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250626 08:23:51.639073 1996 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.025s sys 0.052s
W20250626 08:23:51.693703 1996 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s user 0.026s sys 0.052s
I20250626 08:23:51.695211 3566 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:23:51.695868 2000 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.011s sys 0.020s
I20250626 08:23:51.696877 1848 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:51.598033 4188 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.738919 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2030) took 1085 ms. Trace:
I20250626 08:23:51.739372 1996 rpcz_store.cc:276] 0626 08:23:50.655084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.655235 (+ 151us) service_pool.cc:225] Handling call
0626 08:23:50.671177 (+ 15942us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:50.694793 (+ 23616us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:50.748621 (+ 53828us) log.cc:844] Serialized 14 byte log entry
0626 08:23:51.738582 (+989961us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:51.738652 (+ 70us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:51.738698 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.738763 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:51.738863 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43667,"prepare.queue_time_us":47524,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":883194,"threads_started":1}
I20250626 08:23:51.682715 3813 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.692950 3668 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:51.692139 4056 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:23:51.676622 4271 consensus_queue.cc:1046] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20250626 08:23:51.674813 4270 consensus_queue.cc:1046] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
W20250626 08:23:51.761932 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.742805 4211 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
W20250626 08:23:51.680537 2582 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.008s sys 0.037s
W20250626 08:23:51.770689 2582 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s user 0.008s sys 0.038s
I20250626 08:23:51.750408 4193 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250626 08:23:51.780987 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 89107.4us
I20250626 08:23:51.781872 1967 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:51.765233 4195 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:51.787775 1844 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.000s sys 0.020s
W20250626 08:23:51.800256 1844 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s user 0.000s sys 0.021s
W20250626 08:23:51.806250 1928 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.004s sys 0.033s
I20250626 08:23:51.771788 4078 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:51.767007 4334 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:23:51.811096 3566 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250626 08:23:51.812973 1925 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.017s sys 0.022s
W20250626 08:23:51.813730 1928 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.004s sys 0.033s
W20250626 08:23:51.779208 3952 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.008s sys 0.014s
W20250626 08:23:51.818022 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:51.783751 4193 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:51.784257 4211 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:51.840466 1925 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.600s user 0.017s sys 0.026s
I20250626 08:23:51.805006 4262 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:51.804615 4056 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:51.858134 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.862985 2586 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.020s sys 0.015s
I20250626 08:23:51.862890 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:51.863688 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:51.862691 1845 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.000s sys 0.035s
W20250626 08:23:51.898594 1845 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.000s sys 0.035s
W20250626 08:23:51.899719 2586 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.022s sys 0.016s
I20250626 08:23:51.856076 3976 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:23:51.864889 4215 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.007s sys 0.033s
W20250626 08:23:51.913101 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 131307us
I20250626 08:23:51.854435 2580 raft_consensus.cc:2366] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
W20250626 08:23:51.855834 3952 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s user 0.008s sys 0.014s
I20250626 08:23:51.860030 4193 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:51.977687 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.001s sys 0.044s
I20250626 08:23:51.805191 4164 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.905331 3936 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:51.915678 4109 consensus_meta.cc:287] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.592s user 0.010s sys 0.001s
I20250626 08:23:51.916379 3781 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.916926 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 448) took 1525 ms. Trace:
I20250626 08:23:51.918120 4307 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:51.927150 4315 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 1.117s user 0.005s sys 0.000s
I20250626 08:23:51.928284 3813 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:51.930534 4188 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:51.930629 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1671) took 1134 ms. Trace:
I20250626 08:23:52.011704 1843 rpcz_store.cc:276] 0626 08:23:50.796376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.959076 (+162700us) service_pool.cc:225] Handling call
0626 08:23:50.998810 (+ 39734us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.110031 (+111221us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.930441 (+820410us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b54001f0250. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:51.930573 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90595968}
I20250626 08:23:51.934262 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:52.013262 2587 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:51.935519 4117 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:51.937938 4211 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:51.939199 4027 raft_consensus.cc:3053] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:23:51.942121 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2023) took 1513 ms. Trace:
I20250626 08:23:51.818827 4334 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:52.019356 2002 rpcz_store.cc:276] 0626 08:23:50.428631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.428891 (+ 260us) service_pool.cc:225] Handling call
0626 08:23:50.468136 (+ 39245us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.110640 (+642504us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.941841 (+831201us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54001658d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:51.942052 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1212737408}
I20250626 08:23:52.010720 4307 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:51.946292 1920 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.018s sys 0.015s
W20250626 08:23:52.033481 1920 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.721s user 0.018s sys 0.015s
W20250626 08:23:51.955046 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.034708 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:51.978426 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.670s user 0.001s sys 0.044s
I20250626 08:23:51.814380 3844 consensus_queue.cc:1046] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
W20250626 08:23:51.913125 4215 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.007s sys 0.035s
W20250626 08:23:51.983070 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.016s sys 0.005s
I20250626 08:23:51.840314 3668 consensus_queue.cc:1035] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20250626 08:23:51.983768 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1709) took 1147 ms. Trace:
I20250626 08:23:52.042037 1849 rpcz_store.cc:276] 0626 08:23:50.841423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.233713 (+392290us) service_pool.cc:225] Handling call
0626 08:23:51.234048 (+ 335us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.325056 (+ 91008us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.983462 (+658406us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b540031f1d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:51.983706 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207915776}
W20250626 08:23:52.035189 1995 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.010s sys 0.026s
I20250626 08:23:51.841051 4213 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:51.950026 4156 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.022s sys 0.035s
W20250626 08:23:52.000771 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46905.1us
I20250626 08:23:52.003300 1921 rpcz_store.cc:276] 0626 08:23:50.391815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.432286 (+ 40471us) service_pool.cc:225] Handling call
0626 08:23:50.450067 (+ 17781us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:50.619007 (+168940us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:23:50.619318 (+ 311us) log.cc:844] Serialized 14 byte log entry
0626 08:23:51.461895 (+842577us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:51.461967 (+ 72us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:51.462950 (+ 983us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.916719 (+453769us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400002154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:51.916761 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:51.916873 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40052,"prepare.queue_time_us":549,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":647,"raft.queue_time_us":37811,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":20597,"spinlock_wait_cycles":387866496,"thread_start_us":833214,"threads_started":2,"wal-append.queue_time_us":535385}
I20250626 08:23:51.978811 4223 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:51.896906 4262 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.005595 1993 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:52.008505 4109 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:52.019682 1841 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.015s sys 0.018s
W20250626 08:23:52.063119 1841 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.016s sys 0.018s
I20250626 08:23:52.019959 1843 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250626 08:23:52.015029 1999 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.003s sys 0.031s
W20250626 08:23:52.063889 1999 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.003s sys 0.031s
W20250626 08:23:52.037232 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 739093us
I20250626 08:23:51.982595 4183 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:52.063707 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:52.017529 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.014s sys 0.016s
W20250626 08:23:52.072625 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.014s sys 0.017s
I20250626 08:23:52.072641 1847 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:52.073794 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 608101us
W20250626 08:23:52.020622 1919 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.037s sys 0.021s
W20250626 08:23:51.983546 2581 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.000s sys 0.021s
W20250626 08:23:52.075273 2581 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.000s sys 0.021s
W20250626 08:23:52.075912 1919 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.038s sys 0.021s
W20250626 08:23:52.043056 1995 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.010s sys 0.026s
W20250626 08:23:52.034777 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.002s sys 0.029s
I20250626 08:23:52.030413 4164 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:52.019276 4027 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:23:51.945493 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 82703.5us
I20250626 08:23:52.067843 2550 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:52.142946 4027 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:52.143841 3813 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:52.144178 4027 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:52.144356 3813 raft_consensus.cc:695] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:52.039749 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.016s sys 0.005s
W20250626 08:23:52.146420 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 703771us
W20250626 08:23:52.147589 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.149401 4117 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:52.149773 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:52.150827 3781 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:52.150951 1927 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:52.152056 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:52.152592 1927 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:52.155433 4213 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:52.155889 4334 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.157100 4334 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:23:52.158835 4062 consensus_meta.cc:287] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.001s
I20250626 08:23:52.159310 4062 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:52.160552 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.160739 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1708) took 1324 ms. Trace:
I20250626 08:23:52.161231 1845 rpcz_store.cc:276] 0626 08:23:50.835943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.196409 (+360466us) service_pool.cc:225] Handling call
0626 08:23:51.290355 (+ 93946us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.315259 (+ 24904us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.315294 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.315764 (+ 470us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.899217 (+583453us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:51.899276 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.899373 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:52.160430 (+261057us) raft_consensus.cc:1683] finished
0626 08:23:52.160460 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.160652 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9529,"raft.queue_time_us":18717,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":582971,"threads_started":1,"wal-append.queue_time_us":755922}
I20250626 08:23:52.163727 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1679) took 1347 ms. Trace:
I20250626 08:23:52.164079 1844 rpcz_store.cc:276] 0626 08:23:50.816623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.119341 (+302718us) service_pool.cc:225] Handling call
0626 08:23:51.183228 (+ 63887us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.196507 (+ 13279us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:51.221190 (+ 24683us) log.cc:844] Serialized 14 byte log entry
0626 08:23:51.806778 (+585588us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.818113 (+ 11335us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.834208 (+ 16095us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:51.834259 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.834361 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:52.163544 (+329183us) raft_consensus.cc:1683] finished
0626 08:23:52.163562 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.163675 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61529,"prepare.queue_time_us":3240,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":5354,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":35348,"thread_start_us":579178,"threads_started":1,"wal-append.queue_time_us":811958}
W20250626 08:23:52.170357 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.079406 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.002s sys 0.030s
I20250626 08:23:52.062086 4223 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:52.168884 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:52.204972 2583 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:52.137320 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:52.143353 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 75597.7us
W20250626 08:23:52.046957 4156 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s user 0.023s sys 0.035s
I20250626 08:23:52.145316 3813 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:52.210177 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.014s sys 0.029s
W20250626 08:23:52.210721 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.014s sys 0.029s
W20250626 08:23:52.178794 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 873474us
W20250626 08:23:52.185194 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.044076 4056 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:52.161487 4062 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:52.215698 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:52.218150 1850 raft_consensus.cc:1238] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250626 08:23:52.218240 1842 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.034s sys 0.010s
W20250626 08:23:52.220984 1842 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.531s user 0.035s sys 0.010s
W20250626 08:23:52.223031 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 538186us
W20250626 08:23:52.155452 4189 consensus_meta.cc:287] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.711s user 0.010s sys 0.000s
I20250626 08:23:52.235743 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:52.236835 2584 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:52.253346 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:52.258358 1918 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:52.266628 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:52.267493 1921 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:52.272610 1814 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:52.273881 3566 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:52.274389 3566 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:52.274717 3566 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:52.275805 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.293325 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 150332us
W20250626 08:23:52.319612 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.333164 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1877) took 1054 ms. Trace:
I20250626 08:23:52.361166 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:52.333624 1920 rpcz_store.cc:276] 0626 08:23:51.278701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.279044 (+ 343us) service_pool.cc:225] Handling call
0626 08:23:51.292517 (+ 13473us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.292713 (+ 196us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:52.034013 (+741300us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.332834 (+298821us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:52.332885 (+ 51us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:52.332930 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.332993 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.333110 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57918,"prepare.queue_time_us":849502,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"thread_start_us":962199,"threads_started":2}
W20250626 08:23:52.335350 1923 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.027s sys 0.101s
W20250626 08:23:52.369053 1923 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.028s sys 0.101s
W20250626 08:23:52.408797 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 345156us
W20250626 08:23:52.417052 4331 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.501s user 0.000s sys 0.001s
W20250626 08:23:52.419404 4337 log.cc:927] Time spent T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.515s user 0.002s sys 0.002s
I20250626 08:23:52.423789 4259 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:52.335265 3994 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:52.429070 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1875) took 1318 ms. Trace:
I20250626 08:23:52.450717 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:52.338450 4189 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:52.453744 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.453744 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:52.456393 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:23:52.369665 3952 raft_consensus.cc:2762] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250626 08:23:52.467711 1920 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 6
I20250626 08:23:52.446177 1925 rpcz_store.cc:276] 0626 08:23:51.110977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.239423 (+128446us) service_pool.cc:225] Handling call
0626 08:23:51.239734 (+ 311us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.239997 (+ 263us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:23:51.901019 (+661022us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.428662 (+527643us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:52.428730 (+ 68us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.428786 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.428867 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.429003 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2718,"prepare.queue_time_us":806908,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":37460,"thread_start_us":1003747,"threads_started":2}
I20250626 08:23:52.399969 4117 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:52.335419 3851 consensus_queue.cc:1046] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20250626 08:23:52.425055 3668 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:52.454355 3994 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:52.400279 4062 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20250626 08:23:52.451277 4262 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:52.470312 3791 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:23:52.367791 4335 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
I20250626 08:23:52.517431 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:52.523456 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 286139us
W20250626 08:23:52.542189 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.548616 4259 raft_consensus.cc:695] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:23:52.562732 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.603997 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1712) took 1031 ms. Trace:
I20250626 08:23:52.605037 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1880) took 1287 ms. Trace:
I20250626 08:23:52.656014 1924 rpcz_store.cc:276] 0626 08:23:51.318128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.443359 (+125231us) service_pool.cc:225] Handling call
0626 08:23:51.450494 (+ 7135us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.600212 (+149718us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.604776 (+1004564us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400368dd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:52.604970 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297836416}
I20250626 08:23:52.607594 4264 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:52.613111 4324 log.cc:927] Time spent T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.754s user 0.000s sys 0.004s
W20250626 08:23:52.619031 4332 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.837s user 0.002s sys 0.000s
W20250626 08:23:52.624349 4330 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 1.118s user 0.000s sys 0.001s
I20250626 08:23:52.626675 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 333) took 1367 ms. Trace:
I20250626 08:23:52.632589 3982 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.635761 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1710) took 1131 ms. Trace:
I20250626 08:23:52.515749 3994 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:52.638165 2580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 625) took 1266 ms. Trace:
W20250626 08:23:52.639402 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 330) took 2655 ms (client timeout 1882 ms). Trace:
I20250626 08:23:52.658149 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:52.663414 1925 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:52.484922 4189 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:52.655927 1850 rpcz_store.cc:276] 0626 08:23:51.572567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.580840 (+ 8273us) service_pool.cc:225] Handling call
0626 08:23:51.594383 (+ 13543us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:52.218870 (+624487us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.603694 (+384824us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b5400256cd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:52.603924 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1434003200}
W20250626 08:23:52.672904 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 419604us
I20250626 08:23:52.683365 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:52.481201 3781 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.176s
I20250626 08:23:52.637122 3976 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:52.731979 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:52.455695 4222 consensus_queue.cc:1046] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.198s
I20250626 08:23:52.700466 1846 rpcz_store.cc:276] 0626 08:23:51.503949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.548515 (+ 44566us) service_pool.cc:225] Handling call
0626 08:23:51.568747 (+ 20232us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.612125 (+ 43378us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:23:51.677043 (+ 64918us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.011804 (+334761us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:52.011862 (+ 58us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:52.011916 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.634585 (+622669us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b1800343960. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719
0626 08:23:52.635444 (+ 859us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b540032ebd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:52.635502 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.635682 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":65121,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2051,"raft.queue_time_us":4642,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":3924,"spinlock_wait_cycles":103567488,"thread_start_us":266267,"threads_started":1}
I20250626 08:23:52.648627 4183 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:52.517109 4117 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.516291 3668 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.458228 4223 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:52.915695 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:52.885962 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.746320 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.959506 1924 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
W20250626 08:23:52.814340 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.961071 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:52.932606 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.548707 3566 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:52.911386 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 624) took 1542 ms. Trace:
I20250626 08:23:52.955403 1998 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:52.738050 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:52.970964 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:52.971431 1996 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:52.972043 1998 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:52.974961 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.913004 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:52.958720 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:52.982128 3566 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:52.915206 4346 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.693s user 0.000s sys 0.001s
I20250626 08:23:53.023429 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:52.886950 4038 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.290s user 0.011s sys 0.001s
W20250626 08:23:52.914317 4351 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.266s user 0.000s sys 0.000s
W20250626 08:23:52.914856 4349 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.575s user 0.000s sys 0.000s
W20250626 08:23:52.915849 4354 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.263s user 0.000s sys 0.000s
I20250626 08:23:52.960520 3976 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:52.913379 4357 log.cc:927] Time spent T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.456s user 0.000s sys 0.001s
W20250626 08:23:52.913591 4347 log.cc:927] Time spent T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.486s user 0.001s sys 0.000s
I20250626 08:23:52.966220 4211 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:52.978593 3973 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:52.684862 3791 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:53.110303 3973 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:52.964519 4343 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.754s user 0.000s sys 0.001s
I20250626 08:23:52.699927 3936 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:53.112943 3973 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:53.112505 3791 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:52.911893 4119 raft_consensus.cc:3053] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:53.113523 3973 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.113736 4119 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:53.033267 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:52.965359 2581 rpcz_store.cc:276] 0626 08:23:51.368539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.369416 (+ 877us) service_pool.cc:225] Handling call
0626 08:23:51.369726 (+ 310us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.370114 (+ 388us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.370140 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.370685 (+ 545us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.075614 (+704929us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.075653 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.075711 (+ 58us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:52.911027 (+835316us) raft_consensus.cc:1683] finished
0626 08:23:52.911062 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:52.911255 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6177,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":704569,"threads_started":1,"wal-append.queue_time_us":1101172}
I20250626 08:23:52.876386 4011 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:52.858690 4109 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:53.098687 2551 connection.cc:232] RPC call timeout handler was delayed by -1.6435841020002044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
I20250626 08:23:52.957789 2580 rpcz_store.cc:276] 0626 08:23:51.371394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.371544 (+ 150us) service_pool.cc:225] Handling call
0626 08:23:52.637951 (+1266407us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540044d754. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:52.638108 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729512704}
I20250626 08:23:52.872869 4264 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.106443 1847 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:53.106793 4356 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.107756 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:53.275656 1846 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:23:53.275763 1918 consensus_meta.cc:287] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.009s user 0.010s sys 0.001s
I20250626 08:23:53.110262 4211 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.112825 3936 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.114202 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:53.279234 1842 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:53.032912 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:53.027267 1924 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:52.978286 4335 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.114930 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:53.297214 1849 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:52.683178 1993 rpcz_store.cc:276] 0626 08:23:51.259282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.314322 (+ 55040us) service_pool.cc:225] Handling call
0626 08:23:51.950125 (+635803us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:52.060773 (+110648us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.626368 (+565595us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b5400298950. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:52.626585 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1442937088}
I20250626 08:23:53.115672 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1876) took 1956 ms. Trace:
I20250626 08:23:53.298691 1928 rpcz_store.cc:276] 0626 08:23:51.159099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.242800 (+ 83701us) service_pool.cc:225] Handling call
0626 08:23:51.243110 (+ 310us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.243700 (+ 590us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.243746 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.244208 (+ 462us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.839259 (+595051us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:51.839294 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.839353 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.115459 (+1276106us) raft_consensus.cc:1683] finished
0626 08:23:53.115480 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.115613 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":594368,"threads_started":1,"wal-append.queue_time_us":889862}
I20250626 08:23:53.118301 3976 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.118983 1845 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
W20250626 08:23:52.911075 4338 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.989s user 0.002s sys 0.008s
I20250626 08:23:53.307739 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 651) took 1453 ms. Trace:
I20250626 08:23:53.252758 4161 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:53.125424 2587 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.109s user 0.008s sys 0.000s
I20250626 08:23:53.312948 2587 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:23:53.313603 2587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 558) took 1379 ms. Trace:
I20250626 08:23:53.313997 2587 rpcz_store.cc:276] 0626 08:23:51.933759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.933955 (+ 196us) service_pool.cc:225] Handling call
0626 08:23:53.313535 (+1379580us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:53.131582 4117 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:53.135236 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 644) took 2147 ms. Trace:
I20250626 08:23:52.857703 4344 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:52.965859 3952 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:52.964272 1994 rpcz_store.cc:269] 0626 08:23:49.983945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:49.989223 (+ 5278us) service_pool.cc:225] Handling call
0626 08:23:52.639123 (+2649900us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000d07d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:52.639299 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3556346112}
I20250626 08:23:53.158054 4361 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:53.175665 4164 consensus_meta.cc:287] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.037s user 0.004s sys 0.004s
I20250626 08:23:53.193091 4113 raft_consensus.cc:3053] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:53.199939 3668 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:53.207021 4329 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 1.672s user 0.000s sys 0.013s
I20250626 08:23:53.208912 4062 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:53.228128 4367 log.cc:927] Time spent T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.543s user 0.002s sys 0.000s
I20250626 08:23:53.230237 3988 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:53.246187 1927 consensus_meta.cc:287] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.093s user 0.009s sys 0.002s
I20250626 08:23:53.319558 3886 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:53.257689 2001 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:53.326484 4344 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:53.264884 2583 consensus_meta.cc:287] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.055s user 0.009s sys 0.002s
I20250626 08:23:53.036875 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1889) took 1388 ms. Trace:
W20250626 08:23:53.077744 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:53.084211 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:53.109048 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 452) took 2259 ms. Trace:
W20250626 08:23:53.276739 1921 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.987s user 0.010s sys 0.000s
I20250626 08:23:53.276549 1918 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:52.906414 3982 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 1 FOLLOWER]: Advancing to term 2
W20250626 08:23:53.284993 1893 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.059s sys 0.095s
W20250626 08:23:53.285379 4156 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.057s sys 0.096s
W20250626 08:23:53.286147 4167 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.008s sys 0.036s
W20250626 08:23:53.260071 4307 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.213s user 0.008s sys 0.003s
I20250626 08:23:53.304148 1845 raft_consensus.cc:604] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:23:53.403518 1845 raft_consensus.cc:612] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250626 08:23:53.039176 4193 consensus_meta.cc:287] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.033s user 0.007s sys 0.003s
I20250626 08:23:52.986250 4195 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.037384 4132 raft_consensus.cc:3053] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:53.318622 1890 leader_election.cc:400] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:53.418934 1890 leader_election.cc:403] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250626 08:23:53.326948 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:53.327900 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:53.420974 1928 raft_consensus.cc:2408] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:53.308380 2585 rpcz_store.cc:276] 0626 08:23:51.854341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.855117 (+ 776us) service_pool.cc:225] Handling call
0626 08:23:51.898770 (+ 43653us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.907919 (+ 9149us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.907967 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.912770 (+ 4803us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.400264 (+487494us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.400329 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.400426 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.307417 (+906991us) raft_consensus.cc:1683] finished
0626 08:23:53.307442 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.307660 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7834,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":7283,"thread_start_us":397437,"threads_started":1,"wal-append.queue_time_us":932948}
I20250626 08:23:53.331799 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:53.102362 4038 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:53.335381 1843 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.019s sys 0.031s
I20250626 08:23:53.276491 4078 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:53.439698 1843 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.985s user 0.019s sys 0.031s
W20250626 08:23:53.441326 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 122901us
W20250626 08:23:53.339146 1999 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.003s sys 0.052s
I20250626 08:23:53.320093 2586 rpcz_store.cc:276] 0626 08:23:50.987666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.987813 (+ 147us) service_pool.cc:225] Handling call
0626 08:23:51.036085 (+ 48272us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.070922 (+ 34837us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.070983 (+ 61us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.128383 (+ 57400us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.911117 (+782734us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:51.911171 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.911257 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.134968 (+1223711us) raft_consensus.cc:1683] finished
0626 08:23:53.134997 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.135160 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1600,"raft.queue_time_us":22096,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":13548,"thread_start_us":594406,"threads_started":1,"wal-append.queue_time_us":848634}
W20250626 08:23:53.444087 1999 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.042s user 0.003s sys 0.053s
I20250626 08:23:53.022367 4308 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.303s
I20250626 08:23:53.354173 1842 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:53.259663 3973 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.376753 4062 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:53.475175 1918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1890) took 1826 ms (client timeout 1878 ms). Trace:
W20250626 08:23:53.256422 2551 connection.cc:232] RPC call timeout handler was delayed by -1.0191577059999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20250626 08:23:53.374861 1927 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:53.476181 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 460) took 1327 ms. Trace:
W20250626 08:23:53.475597 1918 rpcz_store.cc:269] 0626 08:23:51.648743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.655793 (+ 7050us) service_pool.cc:225] Handling call
0626 08:23:53.475008 (+1819215us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54001f9354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:53.475118 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1322721024}
I20250626 08:23:53.261365 4011 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.326591 4269 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.133447 4189 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:53.339669 4113 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:53.387146 2583 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:53.541188 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 559) took 1372 ms. Trace:
I20250626 08:23:53.541530 2583 rpcz_store.cc:276] 0626 08:23:52.168412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.168589 (+ 177us) service_pool.cc:225] Handling call
0626 08:23:53.541128 (+1372539us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:53.391280 1921 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:23:53.542649 1921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1895) took 1317 ms (client timeout 1528 ms). Trace:
W20250626 08:23:53.543092 1921 rpcz_store.cc:269] 0626 08:23:52.225059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.228590 (+ 3531us) service_pool.cc:225] Handling call
0626 08:23:53.542578 (+1313988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:53.391654 1923 rpcz_store.cc:276] 0626 08:23:51.648360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.661442 (+ 13082us) service_pool.cc:225] Handling call
0626 08:23:51.681122 (+ 19680us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.741095 (+ 59973us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.741124 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.750856 (+ 9732us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.369429 (+618573us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:52.449875 (+ 80446us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.473928 (+ 24053us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.474035 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.036603 (+562568us) raft_consensus.cc:1683] finished
0626 08:23:53.036630 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.036787 (+ 157us) inbound_call.cc:173] Queueing success response
0626 08:23:53.171720 (+134933us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b18002d5f20. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
Metrics: {"mutex_wait_us":23672,"prepare.queue_time_us":80541,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":148071,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":56064,"spinlock_wait_cycles":16397568,"thread_start_us":550149,"threads_started":1,"wal-append.queue_time_us":865486}
W20250626 08:23:53.393620 1893 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.957s user 0.059s sys 0.096s
W20250626 08:23:53.544255 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.17544e+06us
W20250626 08:23:53.544886 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:53.354049 4083 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.021s sys 0.028s
I20250626 08:23:53.296762 4356 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.039669 4259 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.392244 1841 rpcz_store.cc:276] 0626 08:23:50.849973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.227359 (+377386us) service_pool.cc:225] Handling call
0626 08:23:51.227798 (+ 439us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.289338 (+ 61540us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.289364 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.344387 (+ 55023us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.063480 (+719093us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.063514 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.063570 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.108800 (+1045230us) raft_consensus.cc:1683] finished
0626 08:23:53.108827 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.108980 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78968,"raft.queue_time_us":61266,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"thread_start_us":624256,"threads_started":1,"wal-append.queue_time_us":1006432}
I20250626 08:23:53.311784 4335 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.415238 1998 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:23:53.416334 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 643) took 2516 ms. Trace:
I20250626 08:23:53.330811 4161 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.422430 4307 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.571588 4161 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:53.572088 4307 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:53.573211 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:53.422716 1924 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 6
W20250626 08:23:53.389274 4156 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.019s user 0.057s sys 0.097s
I20250626 08:23:53.439123 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1790) took 1027 ms. Trace:
I20250626 08:23:53.576308 1928 rpcz_store.cc:276] 0626 08:23:52.411465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.300765 (+889300us) service_pool.cc:225] Handling call
0626 08:23:53.439055 (+138290us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:53.449985 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:53.402760 4167 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.973s user 0.008s sys 0.036s
I20250626 08:23:53.386886 4344 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:53.464779 4195 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.582346 4355 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.582943 4355 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.583797 4195 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:53.585004 4355 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:53.394738 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:53.340171 3952 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.561674 1994 raft_consensus.cc:1273] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250626 08:23:53.343720 4361 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.476601 1927 rpcz_store.cc:276] 0626 08:23:52.149151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.151828 (+ 2677us) service_pool.cc:225] Handling call
0626 08:23:53.476122 (+1324294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:53.442615 4193 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.601433 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:53.602051 1927 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:53.339294 4119 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:53.413219 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:53.481421 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:53.612284 1918 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:53.613421 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:53.614118 1918 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:53.532147 2584 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.001s sys 0.032s
W20250626 08:23:53.618242 2584 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.103s user 0.001s sys 0.033s
W20250626 08:23:53.532568 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.058s user 0.006s sys 0.014s
W20250626 08:23:53.619304 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.144s user 0.007s sys 0.014s
W20250626 08:23:53.533141 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.160s user 0.005s sys 0.016s
W20250626 08:23:53.533275 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:53.620218 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.259s user 0.005s sys 0.017s
I20250626 08:23:53.438897 4378 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:53.474296 4251 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.542s user 0.013s sys 0.053s
I20250626 08:23:53.568198 4356 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:53.547778 1920 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.069s user 0.012s sys 0.004s
I20250626 08:23:53.395097 3886 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:53.482420 4188 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.032s user 0.019s sys 0.008s
I20250626 08:23:53.549444 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:53.381601 3988 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.555370 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:53.550031 3851 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.003s sys 0.032s
W20250626 08:23:53.557324 4372 log.cc:927] Time spent T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
I20250626 08:23:53.558162 3973 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:53.558667 4011 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:53.559532 4369 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20250626 08:23:53.560350 4264 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:53.560688 2581 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
W20250626 08:23:53.561198 3994 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.683s user 0.004s sys 0.003s
W20250626 08:23:53.562690 4364 log.cc:927] Time spent T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
W20250626 08:23:53.551874 4083 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s user 0.021s sys 0.028s
I20250626 08:23:53.479173 4215 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:23:53.568207 4335 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:53.568889 4363 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.187s user 0.000s sys 0.000s
I20250626 08:23:53.553253 2582 rpcz_store.cc:276] 0626 08:23:50.900131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.900313 (+ 182us) service_pool.cc:225] Handling call
0626 08:23:50.900671 (+ 358us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:50.955917 (+ 55246us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:50.955952 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:50.956452 (+ 500us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.771525 (+815073us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:51.771585 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:51.772682 (+ 1097us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:52.647689 (+875007us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:53.403208 (+755519us) raft_consensus.cc:1683] finished
0626 08:23:53.403240 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.416235 (+ 12995us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54717,"raft.queue_time_us":56583,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":175,"spinlock_wait_cycles":47346048,"thread_start_us":814540,"threads_started":1,"wal-append.queue_time_us":965016}
I20250626 08:23:53.590423 2580 raft_consensus.cc:2366] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:23:53.471084 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:53.590844 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:53.591517 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:53.591660 2000 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20250626 08:23:53.418418 4132 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:53.471829 4078 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:53.540530 4269 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:53.999358 1843 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.004s sys 0.015s
W20250626 08:23:54.052997 1995 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.005s sys 0.038s
W20250626 08:23:54.053242 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 503946us
I20250626 08:23:54.053613 1920 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:23:54.035619 1846 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.759s user 0.010s sys 0.001s
W20250626 08:23:54.054435 1920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1894) took 1830 ms (client timeout 2176 ms). Trace:
I20250626 08:23:53.657856 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:54.054795 1846 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:23:54.054958 1920 rpcz_store.cc:269] 0626 08:23:52.224538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.414761 (+190223us) service_pool.cc:225] Handling call
0626 08:23:54.054365 (+1639604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:53.631589 1924 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:23:53.733157 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:54.056968 1928 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:53.660877 1998 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:53.596647 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.195s user 0.013s sys 0.043s
I20250626 08:23:54.059376 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1797) took 1506 ms. Trace:
W20250626 08:23:53.687785 1891 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.016s sys 0.021s
I20250626 08:23:54.055580 2000 pending_rounds.cc:85] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 6
I20250626 08:23:54.060073 1816 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:54.056181 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1794) took 1569 ms. Trace:
W20250626 08:23:53.921576 1850 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.008s sys 0.012s
I20250626 08:23:53.640834 1918 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:54.061774 1850 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.014s sys 0.015s
I20250626 08:23:54.062328 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1806) took 1495 ms. Trace:
W20250626 08:23:54.056422 1924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1805) took 1489 ms (client timeout 1916 ms). Trace:
I20250626 08:23:54.062739 1918 rpcz_store.cc:276] 0626 08:23:52.567197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.613092 (+1045895us) service_pool.cc:225] Handling call
0626 08:23:54.062267 (+449175us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:53.836985 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.009s sys 0.014s
W20250626 08:23:54.053259 1843 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.004s sys 0.019s
W20250626 08:23:54.052752 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.016s sys 0.005s
I20250626 08:23:54.056061 1925 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:23:53.899531 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2033) took 3110 ms. Trace:
W20250626 08:23:54.064136 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.016s sys 0.005s
W20250626 08:23:53.631240 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.016s sys 0.022s
W20250626 08:23:54.060011 1891 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.989s user 0.016s sys 0.025s
W20250626 08:23:54.062878 1924 rpcz_store.cc:269] 0626 08:23:52.566851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.331436 (+764585us) service_pool.cc:225] Handling call
0626 08:23:54.056367 (+724931us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:54.064302 1925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1802) took 1498 ms (client timeout 1964 ms). Trace:
W20250626 08:23:53.795195 1996 consensus_meta.cc:287] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.761s user 0.004s sys 0.006s
I20250626 08:23:54.065891 1996 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:54.060653 2000 pending_rounds.cc:107] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 4.7
W20250626 08:23:54.065255 1925 rpcz_store.cc:269] 0626 08:23:52.565691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.417756 (+852065us) service_pool.cc:225] Handling call
0626 08:23:54.064247 (+646491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.059813 1921 rpcz_store.cc:276] 0626 08:23:52.553584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.555034 (+1001450us) service_pool.cc:225] Handling call
0626 08:23:54.059319 (+504285us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:54.055575 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:54.065148 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:54.053452 1995 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.248s user 0.006s sys 0.039s
W20250626 08:23:54.057000 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.008s sys 0.013s
W20250626 08:23:54.064632 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.377s user 0.016s sys 0.025s
W20250626 08:23:54.070006 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.808s user 0.009s sys 0.013s
I20250626 08:23:54.054172 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:23:54.065891 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1791) took 1624 ms. Trace:
I20250626 08:23:54.070746 1928 rpcz_store.cc:276] 0626 08:23:52.441862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.577156 (+1135294us) service_pool.cc:225] Handling call
0626 08:23:54.065795 (+488639us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:54.013938 4374 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.427s user 0.000s sys 0.001s
I20250626 08:23:54.063138 1965 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:53.930353 4334 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.773s user 0.012s sys 0.001s
W20250626 08:23:54.070947 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:54.061576 1923 rpcz_store.cc:276] 0626 08:23:52.487000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.552150 (+1065150us) service_pool.cc:225] Handling call
0626 08:23:54.056119 (+503969us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:53.630887 4027 consensus_meta.cc:287] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.486s user 0.011s sys 0.002s
W20250626 08:23:53.736395 2550 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 1.669s (SENT) [suppressed 6 similar messages]
W20250626 08:23:54.048367 4388 log.cc:927] Time spent T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20250626 08:23:54.067711 2000 raft_consensus.cc:2887] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:54.074035 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2047) took 3087 ms. Trace:
I20250626 08:23:54.077296 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1896) took 1846 ms. Trace:
I20250626 08:23:53.650897 4211 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:54.078209 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1722) took 1740 ms. Trace:
W20250626 08:23:54.078583 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2054) took 1409 ms (client timeout 1878 ms). Trace:
I20250626 08:23:54.008044 3952 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:54.080168 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:53.633049 4251 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.701s user 0.013s sys 0.054s
W20250626 08:23:54.062179 4188 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.621s user 0.029s sys 0.009s
W20250626 08:23:54.062453 3851 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.536s user 0.010s sys 0.035s
W20250626 08:23:54.066838 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.698s user 0.018s sys 0.045s
I20250626 08:23:54.047469 4361 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:53.978394 1994 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 1. Current term is 2. Ops: []
I20250626 08:23:54.086203 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:53.799973 4382 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.808352 4383 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.982139 4119 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:54.063728 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.802s user 0.010s sys 0.015s
I20250626 08:23:53.658655 3976 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:53.679988 4113 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.064292 1997 rpcz_store.cc:276] 0626 08:23:50.789393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:50.797267 (+ 7874us) service_pool.cc:225] Handling call
0626 08:23:50.806085 (+ 8818us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:50.976052 (+169967us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:50.976079 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:50.987213 (+ 11134us) log.cc:844] Serialized 26 byte log entry
0626 08:23:51.919974 (+932761us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b1800005320. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc2f01 00007f4eb9051ccf
0626 08:23:52.292851 (+372877us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:52.292907 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.620048 (+327141us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540018db54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:52.620117 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:53.899230 (+1279113us) raft_consensus.cc:1683] finished
0626 08:23:53.899259 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:53.899449 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27735,"raft.queue_time_us":205,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":10657,"spinlock_wait_cycles":295429376,"thread_start_us":372658,"threads_started":1,"wal-append.queue_time_us":2437226}
I20250626 08:23:53.924686 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.078199 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.090750 1995 rpcz_store.cc:276] 0626 08:23:50.986143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.312172 (+326029us) service_pool.cc:225] Handling call
0626 08:23:51.312719 (+ 547us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:51.364614 (+ 51895us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:51.432895 (+ 68281us) log.cc:844] Serialized 14 byte log entry
0626 08:23:51.567720 (+134825us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b18001f0ba0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:23:52.804649 (+1236929us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.912814 (+108165us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:23:53.229793 (+316979us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b18001f0ba0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:23:54.069661 (+839868us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:54.069721 (+ 60us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:54.069775 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.069849 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.073870 (+ 4021us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":282331,"prepare.queue_time_us":161956,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":1032971,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":182487,"spinlock_wait_cycles":91152512,"thread_start_us":2219823,"threads_started":3,"wal-append.queue_time_us":884952}
W20250626 08:23:54.093865 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:54.095413 1921 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:23:54.058519 4262 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:53.907269 4038 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:54.096954 1996 rpcz_store.cc:269] 0626 08:23:52.658525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.658688 (+ 163us) service_pool.cc:225] Handling call
0626 08:23:54.067609 (+1408921us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.160564 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:53.877609 4380 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:53.733358 4056 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.020s sys 0.018s
I20250626 08:23:54.165095 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:54.098387 1928 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:54.167186 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1792) took 1712 ms. Trace:
I20250626 08:23:53.799746 4290 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:54.103741 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.139420 2578 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:54.134035 4361 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:54.170130 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:54.076200 4239 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.008s sys 0.013s
I20250626 08:23:54.105713 1927 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:54.106240 1816 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:54.108103 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 45035.1us
I20250626 08:23:54.171613 1965 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:54.110025 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1897) took 1776 ms. Trace:
I20250626 08:23:54.172564 1925 rpcz_store.cc:276] 0626 08:23:52.333250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.080588 (+1747338us) service_pool.cc:225] Handling call
0626 08:23:54.095191 (+ 14603us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.109430 (+ 14239us) raft_consensus.cc:1532] Early marking committed up to index 1
0626 08:23:54.109477 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:54.109539 (+ 62us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:54.109644 (+ 105us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.109729 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.109877 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5723,"raft.queue_time_us":9446,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":7149,"spinlock_wait_cycles":24320}
I20250626 08:23:54.115751 2587 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEARNER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:54.116523 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:23:54.104172 4387 log.cc:927] Time spent T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.404s user 0.001s sys 0.000s
I20250626 08:23:54.088708 4159 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:54.085219 4335 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:54.115550 4307 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250626 08:23:54.158838 4262 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:54.084096 4333 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:54.128286 4404 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.121891 3952 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:53.651093 4378 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.140740 1926 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:23:54.125020 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.003s sys 0.020s
I20250626 08:23:54.145100 1890 leader_election.cc:400] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:23:54.219105 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:23:54.220257 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.003s sys 0.020s
I20250626 08:23:54.220314 1890 leader_election.cc:403] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20250626 08:23:54.150187 4167 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.006s sys 0.019s
I20250626 08:23:53.665879 3936 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:54.157125 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1787) took 1746 ms. Trace:
W20250626 08:23:54.157624 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:54.153741 4223 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.028s sys 0.040s
I20250626 08:23:54.164201 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.123185 4308 consensus_queue.cc:1046] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20250626 08:23:54.222916 4223 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.028s sys 0.040s
I20250626 08:23:54.223647 4308 raft_consensus.cc:3053] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:54.224042 4308 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:54.158418 4384 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:54.139982 3851 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:54.147624 4381 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.534s user 0.001s sys 0.001s
I20250626 08:23:54.167677 1928 rpcz_store.cc:276] 0626 08:23:52.454846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.078946 (+1624100us) service_pool.cc:225] Handling call
0626 08:23:54.098067 (+ 19121us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.166920 (+ 68853us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.167128 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.178032 1923 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:54.178656 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.187889 4405 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:54.181881 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.241961 4405 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:54.186344 4391 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
I20250626 08:23:54.244632 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 653) took 1876 ms. Trace:
I20250626 08:23:54.186659 4397 consensus_queue.cc:1046] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250626 08:23:54.186851 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:54.178247 4239 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.917s user 0.008s sys 0.014s
I20250626 08:23:54.246465 4397 raft_consensus.cc:3053] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:54.246557 4239 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:23:54.247344 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.245110 2584 rpcz_store.cc:276] 0626 08:23:52.368206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.368398 (+ 192us) service_pool.cc:225] Handling call
0626 08:23:52.434718 (+ 66320us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:52.435114 (+ 396us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:52.435164 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:52.455134 (+ 19970us) log.cc:844] Serialized 26 byte log entry
0626 08:23:53.618709 (+1163575us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:53.618762 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:53.618864 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.244345 (+625481us) raft_consensus.cc:1683] finished
0626 08:23:54.244372 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.244552 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97651,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":1103308,"threads_started":1,"wal-append.queue_time_us":1337768}
I20250626 08:23:54.249094 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:23:54.252347 4405 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:54.165594 4056 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s user 0.024s sys 0.033s
I20250626 08:23:54.166419 3976 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.166189 1993 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 5
I20250626 08:23:54.068522 3994 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:54.188797 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 82614.2us
I20250626 08:23:54.190239 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.189424 1841 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:54.190591 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:54.146819 4382 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.167275 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2061) took 1139 ms. Trace:
I20250626 08:23:54.221412 4269 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:54.192850 4403 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:54.124692 4132 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.181327 4113 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:54.180629 4038 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.221652 1921 rpcz_store.cc:276] 0626 08:23:52.410387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.078329 (+1667942us) service_pool.cc:225] Handling call
0626 08:23:54.095233 (+ 16904us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.156812 (+ 61579us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.157058 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120064}
I20250626 08:23:54.180300 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 652) took 2239 ms. Trace:
I20250626 08:23:54.225304 1997 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:54.157147 4383 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.238214 4384 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:54.179409 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.357493 2586 raft_consensus.cc:2408] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250626 08:23:54.241202 1995 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:54.216763 4262 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:54.221411 4167 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.608s user 0.006s sys 0.019s
I20250626 08:23:54.242779 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.130728 4251 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250626 08:23:54.246855 4397 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:54.186291 4159 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:54.186388 1927 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:54.253368 4405 raft_consensus.cc:3053] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:54.254345 2582 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:54.185767 4361 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.319828 3973 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.224810 3952 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.269891 4378 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:54.219094 4400 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:54.281877 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:54.401836 1928 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:54.218312 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 481922us
I20250626 08:23:54.351879 4409 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:23:54.187278 4390 consensus_queue.cc:1046] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20250626 08:23:54.225611 4083 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:54.310683 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:54.315670 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2056) took 1656 ms. Trace:
I20250626 08:23:54.310911 1994 rpcz_store.cc:276] 0626 08:23:53.023990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.684488 (+660498us) service_pool.cc:225] Handling call
0626 08:23:53.907049 (+222561us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.140171 (+233122us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.163268 (+ 23097us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:54.314316 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 199582us
I20250626 08:23:54.429071 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.365260 4188 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20250626 08:23:54.430096 1994 raft_consensus.cc:2408] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:23:54.431149 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.431867 1994 raft_consensus.cc:2366] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:23:54.170433 4380 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.432634 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.433383 1994 raft_consensus.cc:2366] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:23:54.381304 4405 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:54.435557 4380 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:54.276221 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:54.213551 4335 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.438256 4335 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:54.259158 3791 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:23:54.351145 1845 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:54.353067 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.438968 3791 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:54.439282 4361 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:54.439894 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.439894 3791 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.441176 4382 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:54.356142 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.452745 1997 raft_consensus.cc:2366] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:23:54.359562 1921 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:23:54.453447 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1798) took 1889 ms. Trace:
I20250626 08:23:54.310649 4027 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:54.346920 4355 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:54.370184 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 203179us
I20250626 08:23:54.406018 2002 rpcz_store.cc:276] 0626 08:23:52.659373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.981803 (+322430us) service_pool.cc:225] Handling call
0626 08:23:53.034820 (+ 53017us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:23:53.035164 (+ 344us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:53.035203 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:23:53.103963 (+ 68760us) log.cc:844] Serialized 52 byte log entry
0626 08:23:54.056592 (+952629us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:54.312058 (+255466us) log.cc:844] Serialized 14 byte log entry
0626 08:23:54.313487 (+ 1429us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.313690 (+ 203us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.313814 (+ 124us) raft_consensus.cc:1683] finished
0626 08:23:54.313827 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.315251 (+ 1424us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37388,"raft.queue_time_us":293561,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":47522,"thread_start_us":728333,"threads_started":2,"wal-append.queue_time_us":1037001}
I20250626 08:23:54.314302 4334 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:54.272878 4392 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:54.319133 4211 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:54.358821 4132 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:54.318704 4078 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:54.346869 4269 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:54.359483 4344 consensus_meta.cc:287] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.777s user 0.008s sys 0.003s
I20250626 08:23:54.258376 3851 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.221246 4404 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:54.381253 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 201138us
I20250626 08:23:54.181041 4290 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.320737 3886 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:54.315040 3988 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:54.353260 4403 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:54.364650 4308 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.273003 3976 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:54.395365 4383 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:54.316298 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 171407us
I20250626 08:23:54.410171 1995 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:54.352669 4038 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:54.414685 4408 log.cc:927] Time spent T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250626 08:23:54.415472 4390 raft_consensus.cc:3053] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:23:54.416020 4083 raft_consensus.cc:695] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:54.416944 3994 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:54.349299 2579 rpcz_store.cc:276] 0626 08:23:51.940989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.941144 (+ 155us) service_pool.cc:225] Handling call
0626 08:23:51.941493 (+ 349us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.941758 (+ 265us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.943525 (+ 1767us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:52.014588 (+ 71063us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.520995 (+506407us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.648386 (+127391us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.685137 (+ 36751us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.159055 (+1473918us) raft_consensus.cc:1683] finished
0626 08:23:54.162595 (+ 3540us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.176167 (+ 13572us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":457408,"threads_started":1,"wal-append.queue_time_us":1374219}
I20250626 08:23:54.382306 4159 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:54.419512 2550 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:54.421357 3813 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:54.378224 4189 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.425295 4397 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:54.389652 3982 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.995s user 0.012s sys 0.000s
I20250626 08:23:54.426946 2582 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:23:54.405756 4406 log.cc:927] Time spent T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20250626 08:23:54.442483 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.443042 3952 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:54.445400 1928 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:54.328694 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:54.450143 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1885) took 3060 ms. Trace:
I20250626 08:23:54.450912 1842 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:23:54.453773 1921 rpcz_store.cc:276] 0626 08:23:52.564271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.347414 (+1783143us) service_pool.cc:225] Handling call
0626 08:23:54.359334 (+ 11920us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:54.453190 (+ 93856us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.453394 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.479172 1923 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:54.497140 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.477623 3918 raft_consensus.cc:1025] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: attempt to promote peer be56082cacb54e9aa8f1a9ee91e70d76: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250626 08:23:54.434796 4405 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.276765 4056 consensus_queue.cc:1035] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20250626 08:23:54.502974 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:55.095228 4083 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.709584 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:55.096776 4394 consensus_queue.cc:1046] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:23:55.097227 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:55.087829 4398 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250626 08:23:55.099053 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2069) took 1367 ms. Trace:
I20250626 08:23:55.099359 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1899) took 1018 ms. Trace:
I20250626 08:23:55.099558 1995 rpcz_store.cc:276] 0626 08:23:53.732069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.156520 (+424451us) service_pool.cc:225] Handling call
0626 08:23:55.098992 (+942472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:55.099776 1923 rpcz_store.cc:276] 0626 08:23:54.080533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.098928 (+ 18395us) service_pool.cc:225] Handling call
0626 08:23:55.099304 (+1000376us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:54.683344 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.100813 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.604157 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1892) took 2386 ms. Trace:
W20250626 08:23:54.527184 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.103252 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1900) took 1015 ms. Trace:
W20250626 08:23:55.103703 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.103821 1928 rpcz_store.cc:276] 0626 08:23:54.088274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.230746 (+142472us) service_pool.cc:225] Handling call
0626 08:23:55.103176 (+872430us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:55.104490 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:55.104952 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:54.882653 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.035s sys 0.064s
W20250626 08:23:55.106722 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.036s sys 0.069s
I20250626 08:23:54.542093 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.503504 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:55.109930 4403 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:55.112473 4403 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:55.095657 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:55.114961 1997 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:55.115810 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:55.107817 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.102277 1920 rpcz_store.cc:276] 0626 08:23:52.217926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.066321 (+1848395us) service_pool.cc:225] Handling call
0626 08:23:54.066660 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:54.077744 (+ 11084us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:54.077773 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:54.078254 (+ 481us) log.cc:844] Serialized 26 byte log entry
0626 08:23:54.319728 (+241474us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:54.319786 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.319873 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.603899 (+284026us) raft_consensus.cc:1683] finished
0626 08:23:54.603924 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.604087 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4790,"raft.queue_time_us":3826,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":11807,"thread_start_us":198615,"threads_started":1,"wal-append.queue_time_us":472432}
I20250626 08:23:55.127475 1920 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:55.132354 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1804) took 2566 ms. Trace:
I20250626 08:23:55.140780 1920 rpcz_store.cc:276] 0626 08:23:52.566499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.126812 (+2560313us) service_pool.cc:225] Handling call
0626 08:23:55.127246 (+ 434us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:55.132021 (+ 4775us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:55.132285 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:55.129837 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:55.146939 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:55.093122 4262 consensus_meta.cc:287] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.718s user 0.008s sys 0.000s
I20250626 08:23:55.102486 1814 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:54.400184 3973 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:54.457670 4400 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:56.186242 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:54.667156 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:55.689746 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.758061 1921 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250626 08:23:55.574456 1847 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:23:56.188611 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1801) took 3623 ms. Trace:
W20250626 08:23:55.479537 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.189396 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:55.509833 1923 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:55.107208 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 687771us
I20250626 08:23:56.193504 2550 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:56.193652 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.189023 1921 rpcz_store.cc:276] 0626 08:23:52.565346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.510801 (+2945455us) service_pool.cc:225] Handling call
0626 08:23:55.757566 (+246765us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.188350 (+430784us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.188555 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.194690 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:55.570554 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:55.508952 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 568) took 1066 ms. Trace:
I20250626 08:23:56.195955 2586 rpcz_store.cc:276] 0626 08:23:54.442093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.442215 (+ 122us) service_pool.cc:225] Handling call
0626 08:23:55.508878 (+1066663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.192802 2002 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:55.510299 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1659) took 1096 ms. Trace:
I20250626 08:23:56.190263 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 661) took 1092 ms. Trace:
W20250626 08:23:56.195295 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:56.188392 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:56.199119 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1669) took 1781 ms (client timeout 2022 ms). Trace:
W20250626 08:23:55.243793 1925 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.004s sys 0.008s
I20250626 08:23:56.199591 2584 rpcz_store.cc:276] 0626 08:23:55.098201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.115451 (+ 17250us) service_pool.cc:225] Handling call
0626 08:23:56.190204 (+1074753us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.200968 2002 rpcz_store.cc:269] 0626 08:23:54.438211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.108745 (+670534us) service_pool.cc:225] Handling call
0626 08:23:56.199043 (+1090298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.610555 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2045) took 3665 ms. Trace:
I20250626 08:23:56.199292 1995 rpcz_store.cc:276] 0626 08:23:54.414389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.100607 (+686218us) service_pool.cc:225] Handling call
0626 08:23:55.510215 (+409608us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.204489 1925 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.741s user 0.017s sys 0.018s
I20250626 08:23:56.186553 1844 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:56.208170 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.248958 1849 raft_consensus.cc:2391] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:55.571125 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:56.250321 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.192116 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:56.251823 1848 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:54.955209 4411 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.511s user 0.001s sys 0.001s
I20250626 08:23:56.271657 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.257762 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 461) took 3282 ms. Trace:
W20250626 08:23:55.062565 4424 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.306s user 0.000s sys 0.000s
W20250626 08:23:56.185536 4113 consensus_meta.cc:287] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.832s user 0.009s sys 0.000s
W20250626 08:23:56.169976 4119 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 2.020s user 0.012s sys 0.000s
W20250626 08:23:54.759070 4413 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.262s user 0.000s sys 0.000s
W20250626 08:23:55.261977 4117 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.012s sys 0.027s
I20250626 08:23:56.193641 1919 rpcz_store.cc:276] 0626 08:23:51.389582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.463452 (+ 73870us) service_pool.cc:225] Handling call
0626 08:23:51.463763 (+ 311us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:23:52.137942 (+674179us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:52.166351 (+ 28409us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:23:52.208954 (+ 42603us) log.cc:844] Serialized 52 byte log entry
0626 08:23:52.656396 (+447442us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:52.875908 (+219512us) log.cc:844] Serialized 14 byte log entry
0626 08:23:52.876188 (+ 280us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:52.876310 (+ 122us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.449882 (+1573572us) raft_consensus.cc:1683] finished
0626 08:23:54.449903 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.450079 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":450718,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":211290,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":13471,"thread_start_us":982916,"threads_started":2,"wal-append.queue_time_us":1154799,"wal-append.run_cpu_time_us":5014,"wal-append.run_wall_time_us":2025338}
I20250626 08:23:55.053754 4420 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:55.997788 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.003s sys 0.006s
I20250626 08:23:55.098842 1994 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:23:55.243505 3791 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.006s sys 0.006s
W20250626 08:23:56.273761 1997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1664) took 1857 ms (client timeout 2124 ms). Trace:
W20250626 08:23:55.063220 4422 log.cc:927] Time spent T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
I20250626 08:23:56.274046 2001 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:56.272783 1920 rpcz_store.cc:276] 0626 08:23:52.975224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.251602 (+3276378us) service_pool.cc:225] Handling call
0626 08:23:56.257175 (+ 5573us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.257382 (+ 207us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.257415 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.257462 (+ 47us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:56.257499 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.257567 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.257707 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.570348 4334 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.276206 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 461) took 1154 ms. Trace:
W20250626 08:23:56.295681 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.900s user 0.003s sys 0.007s
I20250626 08:23:56.277676 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 460) took 1157 ms. Trace:
I20250626 08:23:56.309993 1841 rpcz_store.cc:276] 0626 08:23:55.101184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.101303 (+ 119us) service_pool.cc:225] Handling call
0626 08:23:56.258871 (+1157568us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.113529 1993 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.764s user 0.013s sys 0.000s
I20250626 08:23:54.949326 4417 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:56.281405 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.570338 4384 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:55.288278 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:54.503481 4401 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:55.106485 4390 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:56.032266 4404 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:55.454702 4421 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.000s sys 0.011s
I20250626 08:23:55.575838 4259 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:56.205339 1999 rpcz_store.cc:276] 0626 08:23:50.952582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.340294 (+387712us) service_pool.cc:225] Handling call
0626 08:23:51.376387 (+ 36093us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:51.377109 (+ 722us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:51.377156 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:51.387927 (+ 10771us) log.cc:844] Serialized 26 byte log entry
0626 08:23:52.064363 (+676436us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:23:52.157148 (+ 92785us) log.cc:844] Serialized 14 byte log entry
0626 08:23:53.542040 (+1384892us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:53.542160 (+ 120us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:54.607650 (+1065490us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b540039bfa0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900c2ee 00007f4eb9016e00 00007f4eb9016da8 00007f4eb9016d28 00007f4eb9016ce3 00007f4eb9015f6c 00007f4ebee45ff6 00007f4ebee40848 00007f4eb9052a8b 00007f4eb9055580
0626 08:23:54.610144 (+ 2494us) raft_consensus.cc:1683] finished
0626 08:23:54.610261 (+ 117us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.610456 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":294758,"prepare.queue_time_us":92794,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":717218,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":12537,"spinlock_wait_cycles":21308800,"thread_start_us":1717985,"threads_started":2,"wal-append.queue_time_us":1012276,"wal-append.run_cpu_time_us":46258,"wal-append.run_wall_time_us":3314518}
I20250626 08:23:54.494768 4027 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.290448 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:54.723369 4062 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:56.031688 4132 consensus_meta.cc:287] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.527s user 0.012s sys 0.000s
I20250626 08:23:56.135454 3851 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.292536 1923 raft_consensus.cc:1273] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:56.253316 4213 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.496224 4355 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:56.367015 4444 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:56.367784 4213 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.368209 4444 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:56.295158 4403 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.004s sys 0.021s
W20250626 08:23:56.296613 2583 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.800s user 0.016s sys 0.015s
W20250626 08:23:56.371235 2583 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.876s user 0.017s sys 0.015s
I20250626 08:23:55.742852 4378 consensus_queue.cc:1046] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.546s
I20250626 08:23:56.258402 3886 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:23:56.300714 1997 rpcz_store.cc:269] 0626 08:23:54.416210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.250049 (+1833839us) service_pool.cc:225] Handling call
0626 08:23:56.273696 (+ 23647us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.502355 4211 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:55.575696 4189 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:56.294332 4117 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.580s user 0.029s sys 0.072s
I20250626 08:23:56.302750 1919 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:56.290867 4361 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.009s sys 0.021s
I20250626 08:23:56.275645 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 463) took 1112 ms. Trace:
I20250626 08:23:54.542943 4344 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:54.504101 4269 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:56.306716 1849 rpcz_store.cc:276] 0626 08:23:55.101476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.101590 (+ 114us) service_pool.cc:225] Handling call
0626 08:23:56.256037 (+1154447us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.101177 4083 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.987s user 0.000s sys 0.002s
I20250626 08:23:56.291028 4382 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:56.308722 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 455) took 3208 ms. Trace:
I20250626 08:23:55.575170 3813 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.196295 4262 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.296257 4113 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:54.502017 4392 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.321873 2002 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:23:56.411746 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1657) took 2041 ms (client timeout 2183 ms). Trace:
W20250626 08:23:56.411963 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 219928us
W20250626 08:23:56.412181 2002 rpcz_store.cc:269] 0626 08:23:54.370716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.249605 (+1878889us) service_pool.cc:225] Handling call
0626 08:23:56.411641 (+162036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.413130 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:54.728363 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 555) took 3320 ms. Trace:
W20250626 08:23:56.322853 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.99413e+06us
I20250626 08:23:56.301393 4038 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.301293 4119 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:56.307519 3791 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.863s user 0.017s sys 0.017s
I20250626 08:23:56.330093 4439 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 08:23:56.458418 4439 raft_consensus.cc:695] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:56.459126 4431 log.cc:927] Time spent T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20250626 08:23:56.459964 4392 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.328198 3952 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:56.331001 1842 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1727) took 2166 ms (client timeout 1777 ms). Trace:
W20250626 08:23:56.309340 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1658) took 1905 ms (client timeout 2009 ms). Trace:
I20250626 08:23:56.332037 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1893) took 4112 ms. Trace:
I20250626 08:23:56.332758 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:56.460700 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1799) took 3896 ms. Trace:
I20250626 08:23:56.386729 4390 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:56.461359 1841 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1734) took 1311 ms (client timeout 1505 ms). Trace:
I20250626 08:23:56.461711 4390 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:23:56.461936 1841 rpcz_store.cc:269] 0626 08:23:55.149502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.320985 (+1171483us) service_pool.cc:225] Handling call
0626 08:23:56.461299 (+140314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.462939 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:56.463621 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1735) took 1313 ms. Trace:
I20250626 08:23:56.464030 1841 rpcz_store.cc:276] 0626 08:23:55.149780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.462662 (+1312882us) service_pool.cc:225] Handling call
0626 08:23:56.463571 (+ 909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.461149 1925 rpcz_store.cc:276] 0626 08:23:52.564684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.426352 (+1861668us) service_pool.cc:225] Handling call
0626 08:23:54.426725 (+ 373us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:54.462524 (+ 35799us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:54.462571 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:54.463182 (+ 611us) log.cc:844] Serialized 26 byte log entry
0626 08:23:56.207754 (+1744572us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:56.207813 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.207903 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:56.460497 (+252594us) raft_consensus.cc:1683] finished
0626 08:23:56.460517 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.460642 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":70967,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":49462,"thread_start_us":1779430,"threads_started":2,"wal-append.queue_time_us":1872928}
I20250626 08:23:56.464838 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:56.460582 1996 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 2.
I20250626 08:23:56.465509 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1732) took 1316 ms. Trace:
I20250626 08:23:56.334088 4290 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.335104 4401 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.299161 4420 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.353535 1995 raft_consensus.cc:2366] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:23:56.319479 1993 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:56.413805 2578 rpcz_store.cc:276] 0626 08:23:51.408251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.414449 (+ 6198us) service_pool.cc:225] Handling call
0626 08:23:51.472464 (+ 58015us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.179989 (+2707525us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.726531 (+546542us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b54006066a0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900e2f0 00007f4eb90825e2 00007f4eb907e9c6 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832
0626 08:23:54.728012 (+ 1481us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b5400605ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:54.728273 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6065100928}
W20250626 08:23:56.487473 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2068) took 2756 ms (client timeout 1517 ms). Trace:
W20250626 08:23:56.488106 1993 rpcz_store.cc:269] 0626 08:23:53.731405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.815882 (+ 84477us) service_pool.cc:225] Handling call
0626 08:23:56.487372 (+2671490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:54.097589 1843 rpcz_store.cc:276] 0626 08:23:52.340796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.340955 (+ 159us) service_pool.cc:225] Handling call
0626 08:23:52.414119 (+ 73164us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:52.414327 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:53.440138 (+1025811us) log.cc:844] Serialized 14 byte log entry
0626 08:23:54.077629 (+637491us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:54.077761 (+ 132us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:54.077858 (+ 97us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.077971 (+ 113us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.078137 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7475,"prepare.queue_time_us":1187798,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":57,"raft.queue_time_us":32945,"thread_start_us":1609392,"threads_started":2}
I20250626 08:23:56.365335 3976 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:23:56.336953 4421 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.436s user 0.000s sys 0.013s
I20250626 08:23:56.489382 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:56.492846 4437 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20250626 08:23:56.493304 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1674) took 1500 ms. Trace:
I20250626 08:23:56.493849 1993 rpcz_store.cc:276] 0626 08:23:54.993316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.488996 (+1495680us) service_pool.cc:225] Handling call
0626 08:23:56.493196 (+ 4200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.495137 4420 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.496425 4211 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.497187 4211 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:56.497439 4401 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.490607 4344 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.501258 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 569) took 2048 ms. Trace:
I20250626 08:23:56.501613 2583 rpcz_store.cc:276] 0626 08:23:54.453329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.454159 (+ 830us) service_pool.cc:225] Handling call
0626 08:23:54.454683 (+ 524us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:54.455056 (+ 373us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:54.455081 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:54.494962 (+ 39881us) log.cc:844] Serialized 564 byte log entry
0626 08:23:56.371675 (+1876713us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:56.371711 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.371788 (+ 77us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:56.501050 (+129262us) raft_consensus.cc:1683] finished
0626 08:23:56.501071 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.501199 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1876378,"threads_started":1,"wal-append.queue_time_us":1969408}
I20250626 08:23:56.502977 4262 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.503763 4027 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.394624 4382 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.467351 4119 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.505646 4119 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.506668 1925 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250626 08:23:56.506928 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1661) took 2091 ms (client timeout 1889 ms). Trace:
I20250626 08:23:56.507225 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 466) took 3392 ms. Trace:
W20250626 08:23:56.507269 1996 rpcz_store.cc:269] 0626 08:23:54.415185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.439636 (+ 24451us) service_pool.cc:225] Handling call
0626 08:23:56.506763 (+2067127us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b5400203fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:56.506863 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4645691648}
I20250626 08:23:56.507649 1925 rpcz_store.cc:276] 0626 08:23:53.114532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.464684 (+3350152us) service_pool.cc:225] Handling call
0626 08:23:56.506371 (+ 41687us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.507014 (+ 643us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.507177 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.508463 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.508991 1996 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:56.509096 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 467) took 3394 ms. Trace:
I20250626 08:23:56.332099 4429 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:56.366047 4159 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.798s user 0.010s sys 0.000s
I20250626 08:23:56.335057 2579 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:56.376962 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 464) took 3263 ms. Trace:
W20250626 08:23:56.614606 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 660) took 1517 ms (client timeout 1700 ms). Trace:
I20250626 08:23:56.614500 1919 rpcz_store.cc:276] 0626 08:23:53.113886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.291534 (+3177648us) service_pool.cc:225] Handling call
0626 08:23:56.302575 (+ 11041us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.376648 (+ 74073us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.376898 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.615061 2579 rpcz_store.cc:269] 0626 08:23:55.097273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.287351 (+190078us) service_pool.cc:225] Handling call
0626 08:23:56.614524 (+1327173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.616261 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 468) took 3471 ms. Trace:
I20250626 08:23:56.616612 1919 rpcz_store.cc:276] 0626 08:23:53.144539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.615434 (+3470895us) service_pool.cc:225] Handling call
0626 08:23:56.615769 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.615953 (+ 184us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.615980 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.616019 (+ 39us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:56.616046 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.616094 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.616205 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.378526 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.618274 1997 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:56.371143 4403 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.597s user 0.004s sys 0.021s
I20250626 08:23:56.382823 1844 rpcz_store.cc:276] 0626 08:23:55.146450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.146607 (+ 157us) service_pool.cc:225] Handling call
0626 08:23:56.259048 (+1112441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.370669 4132 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:23:56.382745 4361 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.624s user 0.009s sys 0.022s
I20250626 08:23:56.387511 2580 raft_consensus.cc:1238] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:23:56.388564 1850 rpcz_store.cc:276] 0626 08:23:53.100473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.100600 (+ 127us) service_pool.cc:225] Handling call
0626 08:23:53.108482 (+ 7882us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:53.326803 (+218321us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:53.326833 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:53.327539 (+ 706us) log.cc:844] Serialized 26 byte log entry
0626 08:23:54.062343 (+734804us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:54.062379 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.062441 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:56.308486 (+2246045us) raft_consensus.cc:1683] finished
0626 08:23:56.308510 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.308631 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":217826,"raft.queue_time_us":218003,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":681556,"threads_started":1,"wal-append.queue_time_us":727925}
W20250626 08:23:56.388206 4083 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s user 0.000s sys 0.003s
I20250626 08:23:56.371846 4440 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:56.333273 4417 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.400597 4400 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.401417 4334 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:56.402022 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 657) took 2239 ms. Trace:
I20250626 08:23:56.656531 2582 rpcz_store.cc:276] 0626 08:23:54.162889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.188354 (+ 25465us) service_pool.cc:225] Handling call
0626 08:23:56.401817 (+2213463us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b540053ded4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:56.401964 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72656640}
I20250626 08:23:56.497665 4211 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.413503 2551 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:56.660921 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 247463us
I20250626 08:23:56.414378 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:56.375114 4355 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.387931 4448 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:23:56.450938 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:56.459457 4439 consensus_queue.cc:237] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:56.336270 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:56.465755 1841 rpcz_store.cc:276] 0626 08:23:55.148928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.464610 (+1315682us) service_pool.cc:225] Handling call
0626 08:23:56.465458 (+ 848us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.674593 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1733) took 1525 ms. Trace:
I20250626 08:23:56.675490 1841 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:56.674081 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 414226us
I20250626 08:23:56.467578 2585 raft_consensus.cc:2391] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:56.460577 3952 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:56.460736 1994 rpcz_store.cc:269] 0626 08:23:54.401226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.096960 (+695734us) service_pool.cc:225] Handling call
0626 08:23:56.306758 (+1209798us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.460551 1842 rpcz_store.cc:269] 0626 08:23:54.164647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.164793 (+ 146us) service_pool.cc:225] Handling call
0626 08:23:56.330809 (+2166016us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400106154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:56.330928 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655131520}
W20250626 08:23:56.481108 1995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1647) took 2462 ms (client timeout 2599 ms). Trace:
I20250626 08:23:56.377202 3886 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:56.495155 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.509449 1925 rpcz_store.cc:276] 0626 08:23:53.114867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.508414 (+3393547us) service_pool.cc:225] Handling call
0626 08:23:56.508649 (+ 235us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.508794 (+ 145us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.508827 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.508866 (+ 39us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:56.508893 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.508950 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.509051 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.369822 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1800) took 3804 ms. Trace:
I20250626 08:23:56.568876 2001 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:23:56.633149 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.641157 1997 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:23:56.642608 1996 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:23:56.635828 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 442367us
I20250626 08:23:56.645728 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 469) took 3355 ms. Trace:
I20250626 08:23:56.510843 3813 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:56.490108 3973 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:56.649832 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.450176 4113 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.546249 4450 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:23:56.403107 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 344) took 3379 ms. Trace:
I20250626 08:23:56.633460 4251 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:23:56.660449 1967 connection.cc:232] RPC call timeout handler was delayed by -2.4665993680000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
I20250626 08:23:56.635051 4132 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:56.634820 4401 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:56.460918 1918 rpcz_store.cc:276] 0626 08:23:52.220049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.070550 (+1850501us) service_pool.cc:225] Handling call
0626 08:23:54.070838 (+ 288us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:54.071163 (+ 325us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:54.071190 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:54.080924 (+ 9734us) log.cc:844] Serialized 26 byte log entry
0626 08:23:54.385826 (+304902us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:54.385878 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.385976 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:56.331836 (+1945860us) raft_consensus.cc:1683] finished
0626 08:23:56.331854 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.331978 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3318,"raft.queue_time_us":552,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":9778,"thread_start_us":260207,"threads_started":1,"wal-append.queue_time_us":431420}
I20250626 08:23:56.675285 1849 rpcz_store.cc:276] 0626 08:23:55.149214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.395264 (+1246050us) service_pool.cc:225] Handling call
0626 08:23:56.674505 (+279241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.676743 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:56.678272 1995 rpcz_store.cc:269] 0626 08:23:54.018410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.257114 (+2238704us) service_pool.cc:225] Handling call
0626 08:23:56.481020 (+223906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.681421 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1736) took 1503 ms. Trace:
I20250626 08:23:56.781126 1841 rpcz_store.cc:276] 0626 08:23:55.178536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.674881 (+1496345us) service_pool.cc:225] Handling call
0626 08:23:56.675271 (+ 390us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.681103 (+ 5832us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.681351 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.686915 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1668) took 2269 ms (client timeout 2237 ms). Trace:
W20250626 08:23:56.687198 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 272917us
W20250626 08:23:56.764988 1997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1673) took 1882 ms (client timeout 1963 ms). Trace:
W20250626 08:23:56.687649 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:56.782593 1997 rpcz_store.cc:269] 0626 08:23:54.882192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.378189 (+1495997us) service_pool.cc:225] Handling call
0626 08:23:56.764911 (+386722us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.633946 4159 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.674472 4420 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:56.679373 4262 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:23:56.689189 4400 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:23:56.746187 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 1.64364e+06us
W20250626 08:23:56.773005 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.010s sys 0.013s
I20250626 08:23:56.843822 1995 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20250626 08:23:56.845312 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.866s user 0.010s sys 0.013s
I20250626 08:23:56.845501 1814 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:56.845844 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2065) took 3116 ms. Trace:
I20250626 08:23:56.846248 1995 rpcz_store.cc:276] 0626 08:23:53.729335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.791403 (+3062068us) service_pool.cc:225] Handling call
0626 08:23:56.843557 (+ 52154us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:56.845569 (+ 2012us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.845773 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.847260 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 62947.7us
I20250626 08:23:56.670593 4083 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:56.777459 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 66140.7us
W20250626 08:23:56.780318 2001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 343) took 3757 ms (client timeout 2363 ms). Trace:
W20250626 08:23:56.783594 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.842222 1997 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:56.782124 1999 rpcz_store.cc:269] 0626 08:23:54.417601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.335973 (+1918372us) service_pool.cc:225] Handling call
0626 08:23:56.686845 (+350872us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.784545 1923 rpcz_store.cc:276] 0626 08:23:52.565030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.509236 (+2944206us) service_pool.cc:225] Handling call
0626 08:23:55.509675 (+ 439us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.366711 (+857036us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.369755 (+ 3044us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:56.782074 1965 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.011s sys 0.014s
W20250626 08:23:56.850411 1965 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.891s user 0.012s sys 0.014s
W20250626 08:23:56.850690 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 891589us
W20250626 08:23:56.764863 1967 connection.cc:232] RPC call timeout handler was delayed by -2.3315993680002975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20250626 08:23:56.786099 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:56.851536 1967 connection.cc:232] RPC call timeout handler was delayed by -2.2856993680002233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20250626 08:23:56.851866 1967 connection.cc:232] RPC call timeout handler was delayed by -2.2559993680001753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
W20250626 08:23:56.852149 1967 connection.cc:232] RPC call timeout handler was delayed by -2.1902993680000691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
W20250626 08:23:56.852413 1967 connection.cc:232] RPC call timeout handler was delayed by -2.1470993679999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
W20250626 08:23:56.852691 1967 connection.cc:232] RPC call timeout handler was delayed by -2.1191993679999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20250626 08:23:56.852955 1967 connection.cc:232] RPC call timeout handler was delayed by -1.9346993680001106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20250626 08:23:56.853205 1967 connection.cc:232] RPC call timeout handler was delayed by -1.3604993680000916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2838s for a response
W20250626 08:23:56.853452 1967 connection.cc:232] RPC call timeout handler was delayed by -1.0508993680000458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31820000000000004s for a response
I20250626 08:23:56.780952 1919 rpcz_store.cc:276] 0626 08:23:53.278848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.617449 (+3338601us) service_pool.cc:225] Handling call
0626 08:23:56.617735 (+ 286us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.633228 (+ 15493us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.633260 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.633317 (+ 57us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:56.633364 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.633455 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.633621 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4877,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11596}
I20250626 08:23:56.789060 1996 raft_consensus.cc:1273] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:56.854471 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2063) took 3830 ms. Trace:
I20250626 08:23:56.854804 1996 rpcz_store.cc:276] 0626 08:23:53.024469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.788266 (+3763797us) service_pool.cc:225] Handling call
0626 08:23:56.788638 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.854246 (+ 65608us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.854422 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.855571 1967 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:56.855919 1996 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 5. Ops: []
I20250626 08:23:56.850131 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2067) took 3119 ms. Trace:
I20250626 08:23:56.856493 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1646) took 2877 ms. Trace:
I20250626 08:23:56.856940 1996 rpcz_store.cc:276] 0626 08:23:53.978734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.855479 (+2876745us) service_pool.cc:225] Handling call
0626 08:23:56.855747 (+ 268us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.856283 (+ 536us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.856448 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.622884 4429 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:56.849421 2001 rpcz_store.cc:269] 0626 08:23:53.023209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.380069 (+356860us) service_pool.cc:225] Handling call
0626 08:23:56.271410 (+2891341us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:23:56.780230 (+508820us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17430272}
I20250626 08:23:56.725524 4334 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:56.859313 1996 raft_consensus.cc:1238] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 1. Current term is 3. Ops: []
I20250626 08:23:56.895774 4334 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:56.896234 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1650) took 2801 ms. Trace:
I20250626 08:23:56.896739 1996 rpcz_store.cc:276] 0626 08:23:54.095038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.857785 (+2762747us) service_pool.cc:225] Handling call
0626 08:23:56.859083 (+ 1298us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.895926 (+ 36843us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.896168 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.856320 1997 rpcz_store.cc:276] 0626 08:23:53.730713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.841656 (+3110943us) service_pool.cc:225] Handling call
0626 08:23:56.842025 (+ 369us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.849887 (+ 7862us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.850082 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.898830 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:56.889225 1923 raft_consensus.cc:1273] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250626 08:23:56.900327 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1901) took 2790 ms. Trace:
I20250626 08:23:56.900775 1923 rpcz_store.cc:276] 0626 08:23:54.109832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.850456 (+2740624us) service_pool.cc:225] Handling call
0626 08:23:56.850733 (+ 277us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.900022 (+ 49289us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.900263 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.765018 2000 rpcz_store.cc:276] 0626 08:23:53.023498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.467927 (+444429us) service_pool.cc:225] Handling call
0626 08:23:53.541708 (+ 73781us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.105757 (+564049us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.402773 (+2297016us) spinlock_profiling.cc:243] Waited 49.5 ms on lock 0x7b5400270454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:56.403029 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113833728}
W20250626 08:23:56.889271 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 44496.1us
I20250626 08:23:56.893879 2002 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:56.792884 1918 raft_consensus.cc:1238] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:23:56.904561 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.904853 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1898) took 2830 ms. Trace:
I20250626 08:23:56.905242 1918 rpcz_store.cc:276] 0626 08:23:54.074411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.792315 (+2717904us) service_pool.cc:225] Handling call
0626 08:23:56.792656 (+ 341us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.904630 (+111974us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.904800 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:56.907384 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1904) took 2777 ms. Trace:
I20250626 08:23:56.783555 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 470) took 3232 ms. Trace:
I20250626 08:23:56.991138 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:56.908972 1999 raft_consensus.cc:1238] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3. Ops: []
I20250626 08:23:57.020598 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1645) took 3101 ms. Trace:
W20250626 08:23:56.946050 1845 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.001s sys 0.055s
W20250626 08:23:56.933246 1846 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.023s sys 0.030s
I20250626 08:23:56.908749 1919 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:23:56.901234 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:56.765254 1993 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:56.895206 4119 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:23:56.994498 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 140587us
I20250626 08:23:56.995333 4463 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:56.841353 4450 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.908931 1814 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:23:57.007788 2000 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:23:56.843083 4159 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:56.909829 1995 raft_consensus.cc:1238] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:56.656105 4440 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.019928 1925 rpcz_store.cc:276] 0626 08:23:53.550933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.724162 (+3173229us) service_pool.cc:225] Handling call
0626 08:23:56.783039 (+ 58877us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.783208 (+ 169us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.783234 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.783285 (+ 51us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:56.783319 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.783365 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.783492 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.021304 1918 rpcz_store.cc:276] 0626 08:23:54.131142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.906575 (+2775433us) service_pool.cc:225] Handling call
0626 08:23:56.906842 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.907020 (+ 178us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.907055 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.907111 (+ 56us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:56.907145 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.907212 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:56.907325 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:57.180020 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.225380 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1905) took 3082 ms. Trace:
I20250626 08:23:57.006332 4038 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:56.784420 4132 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:57.081780 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:23:57.205694 2551 connection.cc:232] RPC call timeout handler was delayed by -1.5676488250001057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2363s for a response
W20250626 08:23:57.113508 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 265599us
I20250626 08:23:56.765749 4259 raft_consensus.cc:695] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:56.991729 4083 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:56.893980 4420 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:57.079936 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.064893 1923 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:56.897591 4262 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:57.130106 4464 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.140815 4429 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:57.130913 1926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 458) took 4964 ms (client timeout 1669 ms). Trace:
I20250626 08:23:57.172276 3973 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:57.144330 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:57.086664 4452 log.cc:927] Time spent T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20250626 08:23:57.118265 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:57.226552 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1902) took 3112 ms. Trace:
I20250626 08:23:57.226604 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1644) took 3424 ms. Trace:
I20250626 08:23:57.243289 1919 rpcz_store.cc:276] 0626 08:23:54.114254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.854098 (+2739844us) service_pool.cc:225] Handling call
0626 08:23:56.908454 (+ 54356us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.226304 (+317850us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.226500 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.243392 1995 rpcz_store.cc:276] 0626 08:23:53.816780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.846995 (+3030215us) service_pool.cc:225] Handling call
0626 08:23:56.909634 (+ 62639us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.226036 (+316402us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.226240 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:57.230017 1846 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.946s user 0.030s sys 0.056s
I20250626 08:23:57.225881 1918 rpcz_store.cc:276] 0626 08:23:54.143682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.224441 (+3080759us) service_pool.cc:225] Handling call
0626 08:23:57.224811 (+ 370us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.225008 (+ 197us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:57.225045 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.225098 (+ 53us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:57.225141 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.225204 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.225322 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:57.230089 1845 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s user 0.006s sys 0.080s
I20250626 08:23:57.021097 1999 rpcz_store.cc:276] 0626 08:23:53.919633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.850183 (+2930550us) service_pool.cc:225] Handling call
0626 08:23:56.908797 (+ 58614us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.020260 (+111463us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.020531 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.229091 1967 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:57.248699 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 51789.6us
I20250626 08:23:57.235329 1841 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:57.236464 1849 raft_consensus.cc:2391] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:23:57.250079 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.248699 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:57.240314 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.237448 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1903) took 3113 ms. Trace:
I20250626 08:23:57.251550 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.230587 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1672) took 2476 ms. Trace:
W20250626 08:23:57.239364 1926 rpcz_store.cc:269] 0626 08:23:52.148603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:52.149510 (+ 907us) service_pool.cc:225] Handling call
0626 08:23:57.112728 (+4963218us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54002dfad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:57.112892 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4458979072}
W20250626 08:23:57.237332 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 134862us
I20250626 08:23:56.904126 3976 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:57.284402 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:57.289753 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 380933us
W20250626 08:23:57.292943 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.290208 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1731) took 2141 ms. Trace:
W20250626 08:23:57.303002 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.331413 1850 rpcz_store.cc:276] 0626 08:23:55.148618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.644102 (+1495484us) service_pool.cc:225] Handling call
0626 08:23:56.656857 (+ 12755us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.657097 (+ 240us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:56.680711 (+ 23614us) log.cc:844] Serialized 14 byte log entry
0626 08:23:57.289737 (+609026us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.289795 (+ 58us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:57.289855 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.289965 (+ 110us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.290137 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148559,"prepare.queue_time_us":28495,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":4838,"thread_start_us":460025,"threads_started":1}
I20250626 08:23:57.332921 2000 rpcz_store.cc:276] 0626 08:23:54.754702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.902154 (+2147452us) service_pool.cc:225] Handling call
0626 08:23:57.007407 (+105253us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.230353 (+222946us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.230535 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:57.371815 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.375726 1995 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
W20250626 08:23:57.373253 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 124581us
I20250626 08:23:57.370560 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.296159 1923 rpcz_store.cc:276] 0626 08:23:54.123899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.901781 (+2777882us) service_pool.cc:225] Handling call
0626 08:23:57.008190 (+106409us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.237117 (+228927us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.237377 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.374871 1999 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:23:57.376921 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1677) took 1074 ms. Trace:
I20250626 08:23:57.378145 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1678) took 1054 ms. Trace:
I20250626 08:23:57.378636 1999 rpcz_store.cc:276] 0626 08:23:56.324055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.256835 (+932780us) service_pool.cc:225] Handling call
0626 08:23:57.374695 (+117860us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.377886 (+ 3191us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.378093 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.378917 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.379432 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:57.376065 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:56.898283 4334 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:57.085832 4473 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.304s user 0.000s sys 0.000s
W20250626 08:23:57.383020 2549 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.021s sys 0.051s
W20250626 08:23:57.405118 2549 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.028s user 0.021s sys 0.051s
I20250626 08:23:57.380609 1967 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:57.238525 2578 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.004s sys 0.045s
I20250626 08:23:57.378046 1995 rpcz_store.cc:276] 0626 08:23:56.302972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.260909 (+957937us) service_pool.cc:225] Handling call
0626 08:23:57.375383 (+114474us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.376506 (+ 1123us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.376838 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.235329 1993 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:57.399554 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 459) took 2299 ms. Trace:
I20250626 08:23:57.237154 4420 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:57.408527 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.256597 3791 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.426721 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.443416 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39250 (request call id 1090) took 5238 ms. Trace:
I20250626 08:23:57.445901 1919 rpcz_store.cc:276] 0626 08:23:52.184998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.257972 (+5072974us) service_pool.cc:225] Handling call
0626 08:23:57.422798 (+164826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.230654 4450 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:57.234789 4083 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.440183 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.436578 1845 rpcz_store.cc:276] 0626 08:23:55.100250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.100370 (+ 120us) service_pool.cc:225] Handling call
0626 08:23:55.100637 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:55.100818 (+ 181us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:56.285313 (+1184495us) log.cc:844] Serialized 14 byte log entry
0626 08:23:57.398879 (+1113566us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.399004 (+ 125us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:57.399087 (+ 83us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.399224 (+ 137us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.399467 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399366,"prepare.queue_time_us":854667,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"thread_start_us":1420638,"threads_started":3}
I20250626 08:23:57.449651 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:57.450735 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.442734 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.451697 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.452661 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.441622 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.306489 3851 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.045930 4269 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:57.266126 4384 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.306772 4119 raft_consensus.cc:695] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:57.446406 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.459160 1842 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:56.794935 4401 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:57.464800 1893 connection.cc:232] RPC call timeout handler was delayed by -4.606427146000442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20250626 08:23:57.289040 4159 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.146s
I20250626 08:23:57.306967 4461 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:23:57.425786 4211 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.398022 4062 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:57.450110 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.521525 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:56.897429 4400 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.465849 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.322710 4113 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.239886 3973 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.485266 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.537964 1926 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:57.538841 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:57.238432 4464 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.466933 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.238638 4262 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:57.455116 2583 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.010s sys 0.018s
W20250626 08:23:57.562316 2583 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.010s sys 0.018s
I20250626 08:23:57.562994 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 665) took 1561 ms. Trace:
W20250626 08:23:57.520186 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 54780.4us
W20250626 08:23:57.493443 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 262117us
W20250626 08:23:57.564739 1893 connection.cc:232] RPC call timeout handler was delayed by -4.5068518640000548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2599s for a response
W20250626 08:23:57.565315 1893 connection.cc:232] RPC call timeout handler was delayed by -4.140227146000143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
W20250626 08:23:57.565694 1893 connection.cc:232] RPC call timeout handler was delayed by -3.9938518640001348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3169s for a response
W20250626 08:23:57.565989 1893 connection.cc:232] RPC call timeout handler was delayed by -3.9467271460002848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
I20250626 08:23:57.521597 4471 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:57.566298 1893 connection.cc:232] RPC call timeout handler was delayed by -3.7068226910000703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20250626 08:23:57.525705 1845 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250626 08:23:57.573479 1893 connection.cc:232] RPC call timeout handler was delayed by -3.5934226910003417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
W20250626 08:23:57.573997 1893 connection.cc:232] RPC call timeout handler was delayed by -3.4638226910001322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
I20250626 08:23:57.576047 4460 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.577559 4460 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.568400 4483 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.579398 4400 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.579572 4460 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.579237 4483 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.576117 4355 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.581012 4355 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.545710 3952 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:57.583158 4355 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.576556 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.537406 4417 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:57.581627 4483 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:57.529246 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 154181us
I20250626 08:23:57.531208 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.566075 4211 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:57.589198 4211 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.589509 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.590023 1996 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:23:57.590334 1923 raft_consensus.cc:2408] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250626 08:23:57.590580 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1666) took 3173 ms. Trace:
I20250626 08:23:57.591042 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.591169 4211 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.551908 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 664) took 1683 ms. Trace:
I20250626 08:23:57.592777 4253 raft_consensus.cc:3053] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:23:57.593155 4253 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:23:57.591977 2578 rpcz_store.cc:276] 0626 08:23:55.862591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.487827 (+625236us) service_pool.cc:225] Handling call
0626 08:23:56.488324 (+ 497us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.488633 (+ 309us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:23:56.677203 (+188570us) log.cc:844] Serialized 14 byte log entry
0626 08:23:57.452789 (+775586us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.466208 (+ 13419us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:23:57.486308 (+ 20100us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.525071 (+ 38763us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.546047 (+ 20976us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":100742,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":138229,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":901745,"threads_started":2,"wal-append.queue_time_us":889721}
I20250626 08:23:57.594200 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.594012 4253 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.594475 4239 consensus_queue.cc:1035] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:57.595084 4253 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:57.595440 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.595531 3952 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:23:57.596652 4239 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:23:57.459381 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1665) took 3036 ms (client timeout 3169 ms). Trace:
W20250626 08:23:57.597431 1993 rpcz_store.cc:269] 0626 08:23:54.416533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.494785 (+2078252us) service_pool.cc:225] Handling call
0626 08:23:57.453315 (+958530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:57.531647 4461 raft_consensus.cc:695] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:23:57.591598 1923 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:57.598961 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:57.599439 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:57.599581 4475 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20250626 08:23:57.600073 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.600314 4193 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.600651 1923 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:57.601416 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.384346 3976 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.486083 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.531497 4062 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:57.590904 1996 rpcz_store.cc:276] 0626 08:23:54.416893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.897621 (+2480728us) service_pool.cc:225] Handling call
0626 08:23:57.589666 (+692045us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:57.589845 (+ 179us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:57.590362 (+ 517us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.590529 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13669248}
I20250626 08:23:57.490001 4083 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.561820 1841 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:57.606621 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 226074us
W20250626 08:23:57.524402 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 118212us
I20250626 08:23:57.607496 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:57.566435 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.566988 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:23:57.563390 2583 rpcz_store.cc:276] 0626 08:23:56.001271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.502925 (+501654us) service_pool.cc:225] Handling call
0626 08:23:56.503214 (+ 289us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.503412 (+ 198us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:56.650893 (+147481us) log.cc:844] Serialized 14 byte log entry
0626 08:23:57.562644 (+911751us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.562698 (+ 54us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:57.562751 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.562825 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.562941 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19696,"prepare.queue_time_us":86249,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":561356,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":21064,"thread_start_us":1054569,"threads_started":3}
I20250626 08:23:57.608619 1926 raft_consensus.cc:2391] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:57.609015 1967 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:57.609162 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.524421 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:23:57.575557 1893 connection.cc:232] RPC call timeout handler was delayed by -3.0274440600001071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
I20250626 08:23:57.570833 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:57.610733 1918 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:57.586992 3973 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.537746 3886 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:57.587852 4456 log.cc:927] Time spent T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.280s user 0.000s sys 0.001s
I20250626 08:23:57.596805 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.601838 4269 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:57.464403 4420 consensus_queue.cc:1046] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20250626 08:23:57.459772 3791 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:57.615002 2002 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.712s user 0.010s sys 0.000s
I20250626 08:23:57.542953 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.616816 1965 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:57.617368 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.645797 1919 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:57.645996 1923 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:57.646504 1849 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:23:57.646907 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:23:57.621377 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.615332 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:23:57.648742 1893 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:57.647625 2002 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:57.648118 1918 raft_consensus.cc:2408] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:23:57.653466 1814 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:57.603394 3976 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.637126 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:57.576725 4390 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.114s user 0.008s sys 0.003s
I20250626 08:23:57.644956 1842 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:57.231719 4259 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.603375 4382 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.654716 4083 consensus_queue.cc:1046] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250626 08:23:57.682942 1893 leader_election.cc:336] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 2.599s (SENT)
I20250626 08:23:57.683619 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.648519 4269 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.655884 4333 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:57.656203 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1729) took 2508 ms. Trace:
I20250626 08:23:57.684887 1843 rpcz_store.cc:276] 0626 08:23:55.147974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.491290 (+1343316us) service_pool.cc:225] Handling call
0626 08:23:56.501892 (+ 10602us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:56.502191 (+ 299us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.502229 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:56.502815 (+ 586us) log.cc:844] Serialized 26 byte log entry
0626 08:23:56.683179 (+180364us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:56.683233 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:56.683321 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:57.655987 (+972666us) raft_consensus.cc:1683] finished
0626 08:23:57.656007 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.656147 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":179895,"threads_started":1,"wal-append.queue_time_us":560659}
I20250626 08:23:57.656489 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1730) took 2508 ms. Trace:
I20250626 08:23:57.656975 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.657622 4464 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.598964 4461 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:23:57.612015 4471 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:57.616890 4113 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:23:57.621506 4384 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.611264 4062 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:57.658102 4401 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:57.664376 1923 raft_consensus.cc:2374] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7. Re-sending same reply.
I20250626 08:23:57.697089 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.665154 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.702853 1842 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:23:57.702996 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.703574 1918 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:23:57.666625 1926 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:57.696403 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.662525 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:57.686386 1844 rpcz_store.cc:276] 0626 08:23:55.148317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.643268 (+1494951us) service_pool.cc:225] Handling call
0626 08:23:56.655216 (+ 11948us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:56.655493 (+ 277us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.655531 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:56.670573 (+ 15042us) log.cc:844] Serialized 26 byte log entry
0626 08:23:57.211318 (+540745us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:57.211365 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.211452 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:57.656295 (+444843us) raft_consensus.cc:1683] finished
0626 08:23:57.656318 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:57.656439 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":437690,"threads_started":1,"wal-append.queue_time_us":939384}
I20250626 08:23:57.704684 1849 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:23:57.697834 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.705772 1814 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:57.706451 2583 raft_consensus.cc:2391] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:57.706944 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:23:57.708204 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.710023 1814 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:57.702876 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.711345 1923 raft_consensus.cc:2391] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:57.712733 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.713389 1843 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:23:57.661875 4356 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:23:57.662691 3952 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:57.715801 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:57.716740 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.718086 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:57.662321 3973 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.726153 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:57.622956 4119 consensus_queue.cc:237] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:57.798884 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:58.461719 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:58.039655 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 419159us
W20250626 08:23:57.727932 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 166257us
I20250626 08:23:58.464617 1967 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:58.465250 4496 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250626 08:23:58.164552 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.011s sys 0.015s
W20250626 08:23:58.466356 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.808s user 0.015s sys 0.019s
W20250626 08:23:58.467001 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 854293us
I20250626 08:23:58.464114 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:57.725926 3886 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:58.468052 4496 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:23:58.468351 2583 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:58.498502 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:58.499260 2583 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:23:58.500191 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:58.500705 2583 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:23:58.128574 4474 thread.cc:640] apply [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.000s sys 0.025s
W20250626 08:23:58.111639 4361 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.007s sys 0.028s
I20250626 08:23:58.003387 4269 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:57.803105 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.717486 4193 consensus_queue.cc:1046] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.757s
I20250626 08:23:58.469558 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:57.988876 4463 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.750s user 0.004s sys 0.004s
I20250626 08:23:57.744408 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.167289 4486 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:57.743480 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:54.090847 1924 rpcz_store.cc:276] 0626 08:23:52.225454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.071154 (+1845700us) service_pool.cc:225] Handling call
0626 08:23:54.071391 (+ 237us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.071640 (+ 249us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:54.071664 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:54.071692 (+ 28us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:23:54.071710 (+ 18us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:54.071748 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:54.071824 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2182,"spinlock_wait_cycles":10024192}
I20250626 08:23:57.718210 4038 raft_consensus.cc:695] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:23:58.306041 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:58.470822 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.561647 1923 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:23:57.726599 4356 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:58.471386 1849 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:23:57.786067 4453 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:58.471812 1841 raft_consensus.cc:1238] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:58.570021 1997 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
I20250626 08:23:57.690258 4333 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:58.570946 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1671) took 3972 ms. Trace:
W20250626 08:23:58.472191 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:58.571403 1997 rpcz_store.cc:276] 0626 08:23:54.598198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.898296 (+2300098us) service_pool.cc:225] Handling call
0626 08:23:58.569467 (+1671171us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:58.569749 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.570655 (+ 906us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:58.570881 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11544448}
I20250626 08:23:57.798434 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:58.574337 1844 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:23:57.714367 4239 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:58.139262 1846 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.000s sys 0.026s
W20250626 08:23:58.461941 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.789811 2578 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:23:57.793577 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:57.995321 4471 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:57.830045 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:58.595247 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:58.595674 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:57.983592 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 122156us
I20250626 08:23:57.786382 2579 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:23:58.468537 3886 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:58.485221 4470 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:58.005843 3791 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:23:58.461719 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:23:58.537595 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:58.605885 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 634) took 1030 ms. Trace:
I20250626 08:23:58.606266 2583 rpcz_store.cc:276] 0626 08:23:57.575881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.537219 (+961338us) service_pool.cc:225] Handling call
0626 08:23:58.605826 (+ 68607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.462385 3952 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:23:57.825047 4132 consensus_meta.cc:287] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.598s user 0.008s sys 0.001s
W20250626 08:23:58.549924 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:58.608433 1843 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.019s sys 0.014s
W20250626 08:23:58.608747 1843 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s user 0.019s sys 0.014s
I20250626 08:23:58.551406 1926 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:23:58.609602 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1917) took 1005 ms. Trace:
I20250626 08:23:58.610069 1926 rpcz_store.cc:276] 0626 08:23:57.604465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.704638 (+100173us) service_pool.cc:225] Handling call
0626 08:23:58.609514 (+904876us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.099812 4420 consensus_queue.cc:1230] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250626 08:23:57.718730 3976 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:23:58.548951 1919 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:23:58.615856 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.616406 1919 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:58.617121 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:58.529388 4474 thread.cc:606] apply [worker] (thread pool) Time spent starting thread: real 1.003s user 0.004s sys 0.038s
I20250626 08:23:58.555752 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:58.529626 4361 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.941s user 0.014s sys 0.035s
I20250626 08:23:58.552940 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:23:58.630579 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.631755 1926 raft_consensus.cc:2408] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250626 08:23:58.543649 1890 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:58.557593 4193 consensus_queue.cc:1230] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer 548099d81e7f46ac9b2f03d9e5e678a6 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250626 08:23:58.632879 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 89355.6us
I20250626 08:23:57.811157 4495 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:57.701249 4382 consensus_queue.cc:1046] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.842s
W20250626 08:23:58.592375 1846 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.151s user 0.000s sys 0.048s
I20250626 08:23:58.576577 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.592846 1893 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:58.598418 1968 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:23:58.636027 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:23:58.186918 4334 consensus_meta.cc:287] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.739s user 0.007s sys 0.006s
I20250626 08:23:58.605489 3886 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:58.559856 4463 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:23:58.581518 4478 log.cc:927] Time spent T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 1.008s user 0.013s sys 0.025s
W20250626 08:23:58.582202 4083 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.005s sys 0.018s
I20250626 08:23:58.575671 4356 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:23:58.611869 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:58.612504 1814 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:58.584228 4453 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:23:58.602608 4113 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.014s sys 0.014s
I20250626 08:23:58.617610 1919 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:58.683427 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:23:58.648029 4083 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.640s user 0.008s sys 0.021s
I20250626 08:23:58.635376 1923 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:58.612972 4470 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:58.708973 2582 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20250626 08:23:58.644227 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 91407.5us
I20250626 08:23:58.633980 4496 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250626 08:23:58.652530 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 636) took 1076 ms. Trace:
W20250626 08:23:58.637132 4467 log.cc:927] Time spent T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.468s user 0.002s sys 0.000s
I20250626 08:23:58.658391 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:23:58.714147 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 654) took 6307 ms. Trace:
I20250626 08:23:58.682363 4453 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:58.614995 3976 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:23:58.614404 4132 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:58.684247 1919 raft_consensus.cc:2408] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:23:58.628510 1924 raft_consensus.cc:2408] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for term 7 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250626 08:23:58.744200 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 477) took 1104 ms. Trace:
I20250626 08:23:58.744707 1924 rpcz_store.cc:276] 0626 08:23:57.640163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.555428 (+915265us) service_pool.cc:225] Handling call
0626 08:23:58.744142 (+188714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.684767 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 476) took 1044 ms. Trace:
I20250626 08:23:58.746850 1923 rpcz_store.cc:276] 0626 08:23:57.639911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.576049 (+936138us) service_pool.cc:225] Handling call
0626 08:23:58.684693 (+108644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.748636 1923 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:23:58.612637 3952 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:58.686051 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 191195us
I20250626 08:23:58.652496 4464 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:58.686255 1850 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250626 08:23:58.762727 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 345) took 5739 ms. Trace:
I20250626 08:23:58.763262 1998 rpcz_store.cc:276] 0626 08:23:53.023746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.568733 (+544987us) service_pool.cc:225] Handling call
0626 08:23:53.569082 (+ 349us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.057608 (+488526us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:58.762402 (+4704794us) spinlock_profiling.cc:243] Waited 91.5 ms on lock 0x7b54001662d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:58.762650 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210516352}
I20250626 08:23:58.710162 4503 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:58.753120 3973 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:23:58.752331 3976 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:23:58.685428 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 73028.5us
W20250626 08:23:58.685612 4113 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.850s user 0.015s sys 0.014s
I20250626 08:23:58.712430 2583 rpcz_store.cc:276] 0626 08:23:57.576456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.611592 (+1035136us) service_pool.cc:225] Handling call
0626 08:23:58.652428 (+ 40836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.777566 1814 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:58.777748 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:58.767374 4503 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:58.558109 4486 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:58.924762 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:58.643663 4334 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:23:58.719676 4251 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:58.998078 4251 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:58.721422 4481 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250626 08:23:58.706974 4463 raft_consensus.cc:2762] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250626 08:23:58.639992 4038 consensus_queue.cc:237] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:58.723062 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:59.003363 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.003768 4486 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:58.705173 4239 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:59.004810 4239 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.005338 4239 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:58.754932 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:59.010777 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 255982us
I20250626 08:23:59.011737 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:59.013192 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:58.749342 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1918) took 1144 ms. Trace:
I20250626 08:23:59.014063 1923 rpcz_store.cc:276] 0626 08:23:57.604707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.747788 (+1143081us) service_pool.cc:225] Handling call
0626 08:23:58.748206 (+ 418us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.749062 (+ 856us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:58.749278 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:59.015631 1923 pending_rounds.cc:85] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 1
I20250626 08:23:59.016017 1923 pending_rounds.cc:107] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250626 08:23:59.016321 1923 raft_consensus.cc:2887] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:23:59.032801 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:58.762084 4464 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:23:58.778337 2583 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:23:59.099126 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 672) took 1443 ms. Trace:
I20250626 08:23:59.099586 2583 rpcz_store.cc:276] 0626 08:23:57.655605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.777481 (+1121876us) service_pool.cc:225] Handling call
0626 08:23:59.099040 (+321559us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:58.714591 1926 raft_consensus.cc:2391] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:23:59.100728 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1815) took 1649 ms. Trace:
I20250626 08:23:59.101145 1926 rpcz_store.cc:276] 0626 08:23:57.451268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.639510 (+1188242us) service_pool.cc:225] Handling call
0626 08:23:59.100659 (+461149us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:58.638813 4262 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.014s sys 0.031s
I20250626 08:23:59.140761 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:59.147037 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.002s sys 0.015s
W20250626 08:23:59.147709 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.509s user 0.002s sys 0.015s
W20250626 08:23:59.148633 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 555855us
I20250626 08:23:59.150259 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:23:58.924052 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:59.177843 1890 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.196:37733 timed out after 1.545s (SENT) [suppressed 46 similar messages]
I20250626 08:23:58.714605 2581 rpcz_store.cc:276] 0626 08:23:52.401095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.256334 (+855239us) service_pool.cc:225] Handling call
0626 08:23:53.256732 (+ 398us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:54.058135 (+801403us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:58.708118 (+4649983us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54005e74d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:58.708311 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":678259456}
W20250626 08:23:58.929636 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 132133us
I20250626 08:23:59.179488 1965 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:23:59.179533 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:58.950424 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:58.991329 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 196603us
I20250626 08:23:59.184625 1814 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:58.987900 4514 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:23:58.991906 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:58.992708 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:58.992924 4334 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:58.993474 1918 raft_consensus.cc:2391] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:23:58.993726 1998 raft_consensus.cc:3053] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
W20250626 08:23:58.994004 4506 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.308s user 0.001s sys 0.000s
I20250626 08:23:58.923713 4503 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:58.994549 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:58.995666 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:58.997370 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:58.772068 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 194744us
W20250626 08:23:59.052256 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:59.076903 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.154924 2583 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:23:59.157281 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 124588us
W20250626 08:23:58.898910 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 356524us
I20250626 08:23:59.163391 4382 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:23:59.163831 4420 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:23:59.132375 4439 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.011s sys 0.007s
W20250626 08:23:59.169975 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.023s sys 0.019s
W20250626 08:23:59.273370 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.023s sys 0.019s
W20250626 08:23:59.274024 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.275645 2550 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:59.175297 1926 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:23:59.276988 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1924) took 1669 ms. Trace:
I20250626 08:23:59.277410 1926 rpcz_store.cc:276] 0626 08:23:57.607152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.140258 (+1533106us) service_pool.cc:225] Handling call
0626 08:23:59.175043 (+ 34785us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:59.276651 (+101608us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:59.276919 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:23:59.203317 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:59.282819 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.132894 4464 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.138296 4463 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:59.133086 4505 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:59.208556 1998 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:23:59.155504 4262 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.487s user 0.015s sys 0.032s
I20250626 08:23:59.139552 4504 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:23:59.229014 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 638) took 1641 ms. Trace:
I20250626 08:23:59.321130 2581 rpcz_store.cc:276] 0626 08:23:57.587561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.179201 (+1591640us) service_pool.cc:225] Handling call
0626 08:23:59.228922 (+ 49721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:59.265755 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:59.235505 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:59.314208 1842 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.006s sys 0.031s
I20250626 08:23:59.207260 4269 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:23:59.322779 1842 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.031s
I20250626 08:23:59.322976 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:59.323669 1997 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:23:59.324702 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:59.327813 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:23:59.326891 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 51361.1us
I20250626 08:23:59.335687 2550 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:23:59.343740 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:59.314266 1998 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:59.301060 4439 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.138s user 0.012s sys 0.007s
W20250626 08:23:59.299181 4083 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.008s sys 0.017s
I20250626 08:23:59.316148 4463 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.265872 1993 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
W20250626 08:23:59.408932 4512 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.203s user 0.000s sys 0.000s
I20250626 08:23:59.330989 4269 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.332808 4334 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:23:59.315717 4420 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:23:59.425953 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 275726us
W20250626 08:23:59.407441 4083 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.008s sys 0.017s
W20250626 08:23:59.508740 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:23:59.510866 1924 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.054s sys 0.051s
W20250626 08:23:59.573853 3952 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.806s user 0.009s sys 0.002s
I20250626 08:23:59.581096 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:23:59.574465 4508 log.cc:927] Time spent T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.650s user 0.000s sys 0.011s
I20250626 08:23:59.414018 4419 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:23:59.575757 3886 consensus_meta.cc:287] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.921s user 0.010s sys 0.006s
I20250626 08:23:59.575557 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 663) took 4003 ms. Trace:
W20250626 08:23:59.603323 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.574601 4262 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:59.581266 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:59.620462 1919 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.013s sys 0.045s
W20250626 08:23:59.593241 1927 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1793) took 7131 ms (client timeout 3255 ms). Trace:
I20250626 08:23:59.577888 1928 raft_consensus.cc:1273] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:23:59.577224 1998 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:23:59.161924 4361 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s user 0.000s sys 0.012s
I20250626 08:23:59.207645 4495 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.638741 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:23:59.586040 1999 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:59.638586 1967 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:23:59.642568 1994 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.958s user 0.009s sys 0.008s
W20250626 08:23:59.636706 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 579) took 2143 ms (client timeout 2345 ms). Trace:
I20250626 08:23:59.654839 1994 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:23:59.654918 2583 rpcz_store.cc:269] 0626 08:23:57.493416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.140417 (+1647001us) service_pool.cc:225] Handling call
0626 08:23:59.636497 (+496080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:59.637778 2584 rpcz_store.cc:276] 0626 08:23:55.572434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.255672 (+683238us) service_pool.cc:225] Handling call
0626 08:23:56.286918 (+ 31246us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.679830 (+392912us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:56.725000 (+ 45170us) log.cc:844] Serialized 14 byte log entry
0626 08:23:57.208714 (+483714us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:57.208800 (+ 86us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:57.208864 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:57.586045 (+377181us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b18002f6020. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:23:59.575239 (+1989194us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b5400500354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:59.575325 (+ 86us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:59.575489 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":45789,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":4084,"raft.queue_time_us":292487,"spinlock_wait_cycles":1003330560,"thread_start_us":483180,"threads_started":2,"wal-append.queue_time_us":777024}
W20250626 08:23:59.644845 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 666) took 2113 ms (client timeout 2590 ms). Trace:
I20250626 08:23:59.660598 1967 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:23:59.639355 4078 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.053s sys 0.167s
I20250626 08:23:59.656884 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:23:59.656544 2581 rpcz_store.cc:269] 0626 08:23:57.531372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.349855 (+1818483us) service_pool.cc:225] Handling call
0626 08:23:59.644761 (+294906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:23:59.629447 4132 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:59.672113 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:59.650781 4062 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.925s user 0.005s sys 0.016s
W20250626 08:23:59.648204 4038 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.008s sys 0.005s
W20250626 08:23:59.701963 4038 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s user 0.009s sys 0.005s
W20250626 08:23:59.702195 4356 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.028s user 0.010s sys 0.004s
I20250626 08:23:59.606765 4519 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:23:59.613194 4167 raft_consensus.cc:3053] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:23:59.714018 4510 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.690s user 0.001s sys 0.000s
W20250626 08:23:59.669534 4078 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.925s user 0.055s sys 0.167s
W20250626 08:23:59.652845 4448 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.433s user 0.009s sys 0.052s
I20250626 08:23:59.669638 4083 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:59.638139 4269 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.642275 4518 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:59.648829 4334 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:23:59.721386 4334 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.718945 4183 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:23:59.723817 4183 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:59.724663 4183 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:59.686072 4189 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:59.719074 4448 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s user 0.009s sys 0.053s
W20250626 08:23:59.728773 4480 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 2.142s user 0.002s sys 0.007s
W20250626 08:23:59.688656 4251 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.690s user 0.000s sys 0.017s
I20250626 08:23:59.729300 4269 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.729831 4251 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:59.689337 4513 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.587s user 0.000s sys 0.001s
I20250626 08:23:59.646632 4470 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:59.714058 4167 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:23:59.691152 4132 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:23:59.732435 4132 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:23:59.685566 4453 consensus_meta.cc:287] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.941s user 0.011s sys 0.002s
I20250626 08:23:59.732865 4132 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.733217 4453 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:59.735169 4453 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:59.725870 4183 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:23:59.738620 4213 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:59.740463 4482 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 2.162s user 0.018s sys 0.064s
W20250626 08:23:59.744014 4468 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.284s user 0.018s sys 0.052s
W20250626 08:23:59.697997 4532 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20250626 08:23:59.638218 4463 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:23:59.748169 4390 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250626 08:23:59.749001 4405 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.754956 4444 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:59.754980 4440 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:23:59.697115 4333 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.097s user 0.007s sys 0.010s
I20250626 08:23:59.702785 4356 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:59.710099 4239 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.704s user 0.008s sys 0.004s
I20250626 08:23:59.639451 4262 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:23:59.715201 4519 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.726027 4518 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:23:59.743530 4503 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.639511 4486 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:23:59.760280 4463 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.767452 4420 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:23:59.769783 4239 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:59.784132 4239 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:23:59.769925 4519 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:59.770133 4518 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:23:59.732852 4167 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.791425 4167 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:23:59.770251 4262 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:23:59.772086 4464 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.768013 4333 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:59.773288 4486 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:23:59.781250 4503 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:23:59.637928 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.683184 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:23:59.680261 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:23:59.682471 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:23:59.640183 3886 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:23:59.640262 3952 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:23:59.695346 3976 consensus_meta.cc:287] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.917s user 0.010s sys 0.000s
I20250626 08:24:00.022333 2583 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:00.007743 2000 raft_consensus.cc:2366] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6.
W20250626 08:24:00.037475 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.947436 1845 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
W20250626 08:23:59.974829 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:23:59.943189 1849 raft_consensus.cc:1238] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
W20250626 08:23:59.643510 1927 rpcz_store.cc:269] 0626 08:23:52.461549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:53.602794 (+1141245us) service_pool.cc:225] Handling call
0626 08:23:59.425043 (+5822249us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b54001e1c54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:23:59.593139 (+168096us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115701504}
W20250626 08:23:59.940430 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.000s sys 0.010s
W20250626 08:23:59.635021 1924 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.867s user 0.055s sys 0.059s
I20250626 08:23:59.987102 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1922) took 2381 ms. Trace:
I20250626 08:24:00.127053 4397 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.128655 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:00.129112 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 490272us
I20250626 08:23:59.944977 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1717) took 8205 ms. Trace:
I20250626 08:24:00.128834 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:23:59.628036 1844 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.007s sys 0.050s
W20250626 08:23:59.643504 1919 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s user 0.014s sys 0.045s
W20250626 08:23:59.997879 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 337162us
I20250626 08:23:59.947165 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 462) took 4845 ms. Trace:
I20250626 08:24:00.036650 4167 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:00.020663 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1704) took 2361 ms. Trace:
I20250626 08:24:00.038523 4308 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:00.139400 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.924s user 0.002s sys 0.018s
I20250626 08:24:00.021071 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.033515 4356 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:00.130120 1848 rpcz_store.cc:276] 0626 08:23:51.738846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:51.741096 (+ 2250us) service_pool.cc:225] Handling call
0626 08:23:51.760235 (+ 19139us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.252232 (+4491997us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:59.944048 (+3691816us) spinlock_profiling.cc:243] Waited 4.49 s on lock 0x7b540032f0d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:59.944637 (+ 589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10328923520}
I20250626 08:24:00.054085 4251 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:00.032649 4056 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.136924 1847 rpcz_store.cc:276] 0626 08:23:55.101765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.145404 (+ 43639us) service_pool.cc:225] Handling call
0626 08:23:55.145834 (+ 430us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.188757 (+1042923us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:59.946529 (+3757772us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b540031f6d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:23:59.946940 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":984635136}
I20250626 08:24:00.145038 1927 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:00.034540 4132 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.141848 4167 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:00.126441 1923 rpcz_store.cc:276] 0626 08:23:57.605770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.014973 (+1409203us) service_pool.cc:225] Handling call
0626 08:23:59.015400 (+ 427us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:59.016990 (+ 1590us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:59.017032 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:59.017547 (+ 515us) log.cc:844] Serialized 26 byte log entry
0626 08:23:59.330179 (+312632us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:23:59.527854 (+197675us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:59.527968 (+ 114us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:23:59.986231 (+458263us) raft_consensus.cc:1683] finished
0626 08:23:59.986299 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:23:59.986721 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":312055,"threads_started":1,"wal-append.queue_time_us":402016}
I20250626 08:23:59.910815 4495 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:23:59.822293 4269 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:00.166833 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 667) took 2599 ms (client timeout 2803 ms). Trace:
I20250626 08:23:59.802663 4464 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:00.173872 4056 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.141085 1843 rpcz_store.cc:276] 0626 08:23:57.658775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.714138 (+ 55363us) service_pool.cc:225] Handling call
0626 08:23:57.786985 (+ 72847us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:57.787277 (+ 292us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:57.787317 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:57.811768 (+ 24451us) log.cc:844] Serialized 26 byte log entry
0626 08:23:58.609088 (+797320us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:23:58.609127 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:23:58.609194 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:00.020024 (+1410830us) raft_consensus.cc:1683] finished
0626 08:24:00.020074 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.020423 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":783366,"threads_started":1,"wal-append.queue_time_us":939878}
W20250626 08:24:00.172184 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 783229us
W20250626 08:24:00.178942 2581 rpcz_store.cc:269] 0626 08:23:57.561065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.020160 (+2459095us) service_pool.cc:225] Handling call
0626 08:24:00.160673 (+140513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.169299 4167 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:00.194443 1844 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.433s user 0.008s sys 0.060s
I20250626 08:23:59.809294 4518 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.195276 2001 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 4. Ops: []
I20250626 08:24:00.195387 4448 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:00.195945 4448 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:00.196244 1996 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.018s sys 0.051s
I20250626 08:24:00.196404 4448 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:00.196766 1996 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s user 0.018s sys 0.051s
I20250626 08:23:59.716209 4062 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.199435 3952 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:00.206575 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.208897 1893 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.210186 4356 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:23:59.807296 4519 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.205600 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 357) took 1149 ms. Trace:
I20250626 08:24:00.214465 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1742) took 3307 ms. Trace:
I20250626 08:24:00.211325 1998 rpcz_store.cc:276] 0626 08:23:59.021615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.679936 (+658321us) service_pool.cc:225] Handling call
0626 08:24:00.167180 (+487244us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:00.209707 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 640) took 2529 ms (client timeout 2476 ms). Trace:
W20250626 08:24:00.220530 1999 consensus_meta.cc:287] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.549s user 0.010s sys 0.005s
I20250626 08:24:00.221091 1999 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:00.221293 4463 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:00.221940 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 358) took 1200 ms (client timeout 1571 ms). Trace:
W20250626 08:24:00.222370 1999 rpcz_store.cc:269] 0626 08:23:59.021927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.270793 (+248866us) service_pool.cc:225] Handling call
0626 08:24:00.221799 (+951006us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.214911 1850 rpcz_store.cc:276] 0626 08:23:56.906777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.540060 (+633283us) service_pool.cc:225] Handling call
0626 08:23:57.568122 (+ 28062us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:58.774646 (+1206524us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.213390 (+1438744us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540032ebd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.214292 (+ 902us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2561261440}
I20250626 08:24:00.190851 1993 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:24:00.220451 2584 rpcz_store.cc:269] 0626 08:23:57.655985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.661873 (+2005888us) service_pool.cc:225] Handling call
0626 08:24:00.178017 (+516144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.214865 4519 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:00.197860 4518 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:00.228559 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.018s sys 0.005s
I20250626 08:24:00.199836 3886 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:00.234692 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.236280 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1751) took 1524 ms. Trace:
I20250626 08:24:00.236791 1995 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:24:00.237437 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.237726 4500 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:00.240135 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2077) took 1075 ms. Trace:
I20250626 08:24:00.241173 4333 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:00.242568 1965 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.002s sys 0.022s
W20250626 08:24:00.254669 1842 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.016s sys 0.024s
W20250626 08:24:00.257113 1842 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.934s user 0.017s sys 0.025s
W20250626 08:24:00.256311 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.022s user 0.019s sys 0.006s
W20250626 08:24:00.257880 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.02447e+06us
I20250626 08:24:00.259267 1997 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:00.259573 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.260169 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.262477 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.261667 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 643) took 1490 ms. Trace:
W20250626 08:24:00.265022 1965 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.079s user 0.002s sys 0.023s
I20250626 08:24:00.264717 2584 rpcz_store.cc:276] 0626 08:23:58.771056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.259653 (+1488597us) service_pool.cc:225] Handling call
0626 08:24:00.261526 (+ 1873us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:00.266994 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:00.267592 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.267726 1967 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.268926 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.270296 4503 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:00.250013 1993 rpcz_store.cc:276] 0626 08:23:59.155974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.128102 (+972128us) service_pool.cc:225] Handling call
0626 08:24:00.231289 (+103187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.272440 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.274142 3976 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.243947 1844 rpcz_store.cc:276] 0626 08:23:58.711455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.712160 (+ 705us) service_pool.cc:225] Handling call
0626 08:23:58.720437 (+ 8277us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.720628 (+ 191us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:58.720892 (+ 264us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.235557 (+1514665us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.235676 (+ 119us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.235840 (+ 164us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.235980 (+ 140us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.236162 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40055,"prepare.queue_time_us":1357,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":41582,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":269,"thread_start_us":1473914,"threads_started":1}
I20250626 08:24:00.275542 1841 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
W20250626 08:24:00.262782 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.012s sys 0.011s
I20250626 08:24:00.276604 3976 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:00.276859 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.044s user 0.013s sys 0.011s
I20250626 08:24:00.278746 4486 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:00.279089 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.280862 4262 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.283962 4262 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:00.285521 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.007s sys 0.026s
W20250626 08:24:00.286151 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.007s sys 0.027s
I20250626 08:24:00.286669 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.267482 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1814) took 2816 ms. Trace:
W20250626 08:24:00.286908 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 646197us
I20250626 08:24:00.269740 1998 raft_consensus.cc:2374] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5. Re-sending same reply.
W20250626 08:24:00.262398 4109 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.004s sys 0.028s
I20250626 08:24:00.292521 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:00.294164 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:00.294350 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 85625.5us
W20250626 08:24:00.294621 1918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 1916) took 2690 ms (client timeout 2099 ms). Trace:
I20250626 08:24:00.295095 1998 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
W20250626 08:24:00.295038 1918 rpcz_store.cc:269] 0626 08:23:57.604240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.743109 (+138869us) service_pool.cc:225] Handling call
0626 08:24:00.294250 (+2551141us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54002ae9d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.294438 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1007711104}
I20250626 08:24:00.295902 4356 consensus_queue.cc:1046] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:24:00.297955 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 462) took 7318 ms. Trace:
I20250626 08:24:00.300922 4062 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:00.301079 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1803) took 7735 ms. Trace:
W20250626 08:24:00.292915 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.009s sys 0.013s
I20250626 08:24:00.298882 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:00.264235 1926 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.000s sys 0.014s
I20250626 08:24:00.309487 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.301571 1928 rpcz_store.cc:276] 0626 08:23:52.566026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:55.511350 (+2945324us) service_pool.cc:225] Handling call
0626 08:23:59.577504 (+4066154us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:59.643761 (+ 66257us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.300648 (+656887us) spinlock_profiling.cc:243] Waited 4.07 s on lock 0x7b54001e1c50. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.300898 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9350386048}
I20250626 08:24:00.298409 1921 rpcz_store.cc:276] 0626 08:23:52.978914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.248834 (+3269920us) service_pool.cc:225] Handling call
0626 08:23:56.281043 (+ 32209us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.371003 (+ 89960us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.371033 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.371074 (+ 41us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:56.371103 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.297416 (+3926313us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b5400300554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.297503 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.297747 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206396288}
W20250626 08:24:00.327795 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.985s user 0.012s sys 0.013s
I20250626 08:24:00.321177 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:00.312548 2585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 570) took 5858 ms (client timeout 1505 ms). Trace:
W20250626 08:24:00.337047 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 78088.5us
I20250626 08:24:00.343055 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.343547 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 662) took 5243 ms. Trace:
W20250626 08:24:00.343729 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.16197e+06us
W20250626 08:24:00.292543 4109 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.718s user 0.004s sys 0.028s
I20250626 08:24:00.344039 1928 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.345113 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 574) took 2888 ms. Trace:
I20250626 08:24:00.345522 2582 rpcz_store.cc:276] 0626 08:23:57.425733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.448938 (+ 23205us) service_pool.cc:225] Handling call
0626 08:23:58.708522 (+1259584us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.709477 (+ 955us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.313388 (+1603911us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54005e74d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.313647 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2811062528}
I20250626 08:24:00.346462 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.347353 1921 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.348428 4538 consensus_queue.cc:1046] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20250626 08:24:00.302245 4356 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:00.350706 4356 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:24:00.309648 2578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 637) took 2732 ms (client timeout 1545 ms). Trace:
W20250626 08:24:00.351588 2578 rpcz_store.cc:269] 0626 08:23:57.576696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.596565 (+ 19869us) service_pool.cc:225] Handling call
0626 08:24:00.309208 (+2712643us) spinlock_profiling.cc:243] Waited 87.8 ms on lock 0x7b54006149d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.309414 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":201928576}
I20250626 08:24:00.352233 4356 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.353058 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.287794 1919 rpcz_store.cc:276] 0626 08:23:57.451043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.742321 (+1291278us) service_pool.cc:225] Handling call
0626 08:23:58.742733 (+ 412us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.743012 (+ 279us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:23:58.743403 (+ 391us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.250893 (+1507490us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.266934 (+ 16041us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.267070 (+ 136us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.267156 (+ 86us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.267307 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12448,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":6606,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":42338,"thread_start_us":1404341,"threads_started":1,"wal-append.queue_time_us":1478785}
I20250626 08:24:00.355366 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.355993 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1724) took 6193 ms. Trace:
W20250626 08:24:00.356268 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.12078e+06us
I20250626 08:24:00.362283 2584 raft_consensus.cc:2391] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:00.363487 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.364092 2584 raft_consensus.cc:2374] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4. Re-sending same reply.
I20250626 08:24:00.364506 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 678) took 1881 ms. Trace:
I20250626 08:24:00.365095 2584 rpcz_store.cc:276] 0626 08:23:58.483019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.363183 (+1880164us) service_pool.cc:225] Handling call
0626 08:24:00.364461 (+ 1278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.347251 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.301692 4062 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:00.356047 1919 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:00.365999 1926 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s user 0.001s sys 0.016s
I20250626 08:24:00.367214 4361 raft_consensus.cc:3053] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:00.367643 4448 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:00.368729 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 673) took 2707 ms (client timeout 2858 ms). Trace:
W20250626 08:24:00.383081 2581 rpcz_store.cc:269] 0626 08:23:57.660841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.233356 (+2572515us) service_pool.cc:225] Handling call
0626 08:24:00.368662 (+135306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.369298 1998 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:24:00.371529 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 355) took 1350 ms. Trace:
I20250626 08:24:00.371847 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1743) took 3451 ms. Trace:
I20250626 08:24:00.387427 1928 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:24:00.372677 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 80394.5us
I20250626 08:24:00.374011 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1750) took 1663 ms. Trace:
I20250626 08:24:00.356621 1846 rpcz_store.cc:276] 0626 08:23:54.163190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.168468 (+ 5278us) service_pool.cc:225] Handling call
0626 08:23:54.168794 (+ 326us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:57.244819 (+3076025us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:57.244849 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:57.257469 (+ 12620us) log.cc:844] Serialized 26 byte log entry
0626 08:23:58.634858 (+1377389us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:23:58.634901 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.355303 (+1720402us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400033ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.355363 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:00.355424 (+ 61us) raft_consensus.cc:1683] finished
0626 08:24:00.355438 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.355815 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2666,"raft.queue_time_us":63962,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":4856849152,"thread_start_us":2154497,"threads_started":2,"wal-append.queue_time_us":1220954}
I20250626 08:24:00.353678 4356 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.375891 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1908) took 6224 ms. Trace:
W20250626 08:24:00.336113 2585 rpcz_store.cc:269] 0626 08:23:54.453745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.453863 (+ 118us) service_pool.cc:225] Handling call
0626 08:24:00.312007 (+5858144us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400541fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.312307 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3114297728}
I20250626 08:24:00.386219 1845 rpcz_store.cc:276] 0626 08:23:56.920591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.586015 (+665424us) service_pool.cc:225] Handling call
0626 08:23:59.946952 (+2360937us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.136230 (+189278us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.371646 (+235416us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b540031f6d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.371794 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5428599552}
I20250626 08:24:00.389309 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:00.392617 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 103075us
I20250626 08:24:00.344077 2586 rpcz_store.cc:276] 0626 08:23:55.099567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.197463 (+1097896us) service_pool.cc:225] Handling call
0626 08:23:56.282164 (+ 84701us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.399118 (+116954us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:56.590252 (+191134us) log.cc:844] Serialized 14 byte log entry
0626 08:23:56.606081 (+ 15829us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.606156 (+ 75us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:23:56.606212 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.342986 (+3736774us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54005eae54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.343087 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.343346 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":190020,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":8396,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":268410240,"thread_start_us":148446,"threads_started":2,"wal-append.queue_time_us":67561,"wal-append.run_cpu_time_us":14819,"wal-append.run_wall_time_us":1789472}
I20250626 08:24:00.393461 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.392726 4503 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.394208 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.386628 4361 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.400014 1921 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:00.311323 2580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 560) took 6200 ms. Trace:
I20250626 08:24:00.311269 4132 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:00.395689 4448 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:00.389302 1925 rpcz_store.cc:276] 0626 08:23:54.151296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.224520 (+3073224us) service_pool.cc:225] Handling call
0626 08:24:00.374479 (+3149959us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:00.374834 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.375395 (+ 561us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.375426 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.375489 (+ 63us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:00.375528 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.375589 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.375722 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40678144}
I20250626 08:24:00.408723 4503 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.409695 4448 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:00.410260 4448 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:00.410161 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.410845 4448 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:00.411394 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2085) took 1012 ms. Trace:
I20250626 08:24:00.411854 1993 rpcz_store.cc:276] 0626 08:23:59.398644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.273844 (+875200us) service_pool.cc:225] Handling call
0626 08:24:00.409829 (+135985us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:00.411300 (+ 1471us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51002240}
I20250626 08:24:00.410204 2581 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.399698 4545 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20250626 08:24:00.413309 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.396579 3886 consensus_queue.cc:1046] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.018s
W20250626 08:24:00.416312 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.417339 2550 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:00.418594 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.404006 2580 rpcz_store.cc:276] 0626 08:23:54.110941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.113651 (+ 2710us) service_pool.cc:225] Handling call
0626 08:23:54.165508 (+ 51857us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.650126 (+2484618us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.310714 (+3660588us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54003edfd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.311075 (+ 361us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5103269632}
I20250626 08:24:00.422353 1998 raft_consensus.cc:2374] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3. Re-sending same reply.
I20250626 08:24:00.422847 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 353) took 1464 ms. Trace:
I20250626 08:24:00.423233 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.423605 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.424753 1998 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 8: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:00.425015 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1754) took 1241 ms. Trace:
I20250626 08:24:00.426291 3988 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.427014 3988 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:00.395723 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2070) took 6662 ms (client timeout 3156 ms). Trace:
I20250626 08:24:00.430244 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 465) took 7315 ms. Trace:
W20250626 08:24:00.430153 2002 rpcz_store.cc:269] 0626 08:23:53.732707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.412876 (+2680169us) service_pool.cc:225] Handling call
0626 08:24:00.395296 (+3982420us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400298954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.395500 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1104369536}
I20250626 08:24:00.430662 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1662) took 6015 ms. Trace:
I20250626 08:24:00.431334 4397 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:00.431541 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.432536 2002 raft_consensus.cc:2391] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:00.433048 3988 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:00.434387 1997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 359) took 1412 ms (client timeout 1849 ms). Trace:
I20250626 08:24:00.380138 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 676) took 1871 ms. Trace:
I20250626 08:24:00.431144 2001 rpcz_store.cc:276] 0626 08:23:54.415521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.896895 (+2481374us) service_pool.cc:225] Handling call
0626 08:23:59.424449 (+2527554us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:23:59.576114 (+151665us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.195943 (+619829us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.430370 (+234427us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b5400203ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.430573 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429445248}
I20250626 08:24:00.425419 1849 rpcz_store.cc:276] 0626 08:23:59.184009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.224709 (+ 40700us) service_pool.cc:225] Handling call
0626 08:23:59.304572 (+ 79863us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.141845 (+837273us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.424743 (+282898us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b540029d6d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.424955 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1466975488}
I20250626 08:24:00.423341 2000 rpcz_store.cc:276] 0626 08:23:58.958631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.992433 (+ 33802us) service_pool.cc:225] Handling call
0626 08:24:00.422560 (+1430127us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b5400079ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.422754 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1851367808}
I20250626 08:24:00.437165 4503 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:00.384824 1996 rpcz_store.cc:276] 0626 08:23:59.020841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.132496 (+111655us) service_pool.cc:225] Handling call
0626 08:23:59.132958 (+ 462us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:23:59.157056 (+ 24098us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:59.157087 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:23:59.170912 (+ 13825us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.197403 (+1026491us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:00.197489 (+ 86us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.197586 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:00.371309 (+173723us) raft_consensus.cc:1683] finished
0626 08:24:00.371327 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.371444 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23646,"raft.queue_time_us":24159,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":93,"thread_start_us":892142,"threads_started":1,"wal-append.queue_time_us":1138053}
W20250626 08:24:00.404245 2582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 578) took 2938 ms (client timeout 3798 ms). Trace:
W20250626 08:24:00.485460 2582 rpcz_store.cc:269] 0626 08:23:57.465708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.346854 (+2881146us) service_pool.cc:225] Handling call
0626 08:24:00.404174 (+ 57320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.425639 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.433516 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.438907 1965 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.430749 1920 rpcz_store.cc:276] 0626 08:23:53.114235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.303790 (+3189555us) service_pool.cc:225] Handling call
0626 08:23:56.306607 (+ 2817us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:56.329311 (+ 22704us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:23:56.329343 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:23:56.329383 (+ 40us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:23:56.329412 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.429733 (+4100321us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b54001f8954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.429825 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.430050 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51752704}
I20250626 08:24:00.455083 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.456152 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2084) took 1136 ms. Trace:
I20250626 08:24:00.487329 1999 rpcz_store.cc:276] 0626 08:23:59.320054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.327767 (+1007713us) service_pool.cc:225] Handling call
0626 08:24:00.456089 (+128322us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:00.456869 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 235729us
I20250626 08:24:00.421309 4361 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:00.466284 1997 rpcz_store.cc:269] 0626 08:23:59.022241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.325703 (+303462us) service_pool.cc:225] Handling call
0626 08:24:00.434216 (+1108513us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540039ce54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.434336 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070267776}
I20250626 08:24:00.477070 2581 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:00.493278 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.492522 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.495798 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.496986 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.497073 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.497437 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.498201 1920 raft_consensus.cc:2391] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:00.498389 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2081) took 1268 ms. Trace:
I20250626 08:24:00.498600 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.499894 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.500067 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1929) took 2871 ms. Trace:
I20250626 08:24:00.500324 2581 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:45504
I20250626 08:24:00.501135 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.502702 1921 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20250626 08:24:00.503134 1921 pending_rounds.cc:85] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 4
I20250626 08:24:00.502959 4132 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:00.503556 1921 pending_rounds.cc:107] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250626 08:24:00.504504 1921 raft_consensus.cc:2887] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:00.505213 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2080) took 1300 ms. Trace:
I20250626 08:24:00.505739 2002 rpcz_store.cc:276] 0626 08:23:59.204670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.433140 (+1228470us) service_pool.cc:225] Handling call
0626 08:24:00.505116 (+ 71976us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.506379 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.506995 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.498816 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.508018 2002 raft_consensus.cc:2374] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4. Re-sending same reply.
W20250626 08:24:00.509105 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 677) took 2026 ms (client timeout 1700 ms). Trace:
W20250626 08:24:00.518540 2583 rpcz_store.cc:269] 0626 08:23:58.482687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.656630 (+1173943us) service_pool.cc:225] Handling call
0626 08:24:00.508741 (+852111us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54004024d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.508998 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":825222912}
I20250626 08:24:00.520048 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:00.509764 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 57490.6us
W20250626 08:24:00.526862 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1675) took 4254 ms (client timeout 2089 ms). Trace:
I20250626 08:24:00.526960 1816 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:00.527465 1994 rpcz_store.cc:269] 0626 08:23:56.272643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:56.708682 (+436039us) service_pool.cc:225] Handling call
0626 08:24:00.526458 (+3817776us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b540039d354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.526651 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3312567040}
I20250626 08:24:00.528316 3886 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:00.510520 4519 consensus_queue.cc:1035] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20250626 08:24:00.530287 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 573) took 3123 ms (client timeout 1688 ms). Trace:
W20250626 08:24:00.530737 2579 rpcz_store.cc:269] 0626 08:23:57.407196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:57.425887 (+ 18691us) service_pool.cc:225] Handling call
0626 08:24:00.529957 (+3104070us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54005438d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.530119 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269174656}
I20250626 08:24:00.512815 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1752) took 1329 ms. Trace:
I20250626 08:24:00.532938 1841 rpcz_store.cc:276] 0626 08:23:59.183306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.185649 (+ 2343us) service_pool.cc:225] Handling call
0626 08:23:59.225153 (+ 39504us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.276002 (+1050849us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.512377 (+236375us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400210f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.512694 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2414850816}
W20250626 08:24:00.534070 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 94146.9us
I20250626 08:24:00.513975 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 351) took 1558 ms. Trace:
I20250626 08:24:00.534446 4503 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:00.534933 1995 rpcz_store.cc:276] 0626 08:23:58.955156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.955274 (+ 118us) service_pool.cc:225] Handling call
0626 08:23:58.993204 (+ 37930us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.264075 (+1270871us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.513580 (+249505us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002805d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.513879 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2779797760}
I20250626 08:24:00.532354 2579 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:00.510175 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.538614 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 668) took 2972 ms. Trace:
I20250626 08:24:00.538975 2000 raft_consensus.cc:2374] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5. Re-sending same reply.
I20250626 08:24:00.539245 2579 rpcz_store.cc:276] 0626 08:23:57.566061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.531548 (+2965487us) service_pool.cc:225] Handling call
0626 08:24:00.531975 (+ 427us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.538197 (+ 6222us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.538497 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.540539 2000 raft_consensus.cc:1273] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250626 08:24:00.540913 2000 pending_rounds.cc:85] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 3
I20250626 08:24:00.541116 2000 pending_rounds.cc:107] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250626 08:24:00.541335 2000 raft_consensus.cc:2887] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:00.514146 1928 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:00.542237 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 633) took 2972 ms. Trace:
I20250626 08:24:00.542809 2579 rpcz_store.cc:276] 0626 08:23:57.570008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.540688 (+2970680us) service_pool.cc:225] Handling call
0626 08:24:00.541226 (+ 538us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.541612 (+ 386us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.541657 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.541781 (+ 124us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.541858 (+ 77us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.541971 (+ 113us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.542138 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.388839 1842 rpcz_store.cc:276] 0626 08:23:58.711086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.712060 (+ 974us) service_pool.cc:225] Handling call
0626 08:23:58.717275 (+ 5215us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.717455 (+ 180us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:23:58.717759 (+ 304us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.373285 (+1655526us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.373402 (+ 117us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:00.373491 (+ 89us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.373721 (+ 230us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.373949 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":620,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":297057,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":8593,"thread_start_us":1539459,"threads_started":2,"wal-append.queue_time_us":929128}
I20250626 08:24:00.500514 1920 rpcz_store.cc:276] 0626 08:23:57.628743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.498875 (+2870132us) service_pool.cc:225] Handling call
0626 08:24:00.499302 (+ 427us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.499710 (+ 408us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.499738 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.499796 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:00.499835 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.499893 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.499991 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.537793 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 361) took 1515 ms. Trace:
I20250626 08:24:00.537653 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2073) took 1439 ms. Trace:
W20250626 08:24:00.489230 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 72012.9us
I20250626 08:24:00.505414 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1930) took 2868 ms. Trace:
I20250626 08:24:00.498602 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.547376 1921 rpcz_store.cc:276] 0626 08:23:57.637283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.501634 (+2864351us) service_pool.cc:225] Handling call
0626 08:24:00.502190 (+ 556us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.505157 (+ 2967us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.505346 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.546417 1994 rpcz_store.cc:276] 0626 08:23:59.022826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.528493 (+1505667us) service_pool.cc:225] Handling call
0626 08:24:00.528892 (+ 399us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.537315 (+ 8423us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.537359 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.537434 (+ 75us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:00.537530 (+ 96us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.537598 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.537711 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.546661 1995 rpcz_store.cc:276] 0626 08:23:59.098421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.536412 (+1437991us) service_pool.cc:225] Handling call
0626 08:24:00.536882 (+ 470us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.537235 (+ 353us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.537266 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.537308 (+ 42us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.537351 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.537423 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.537566 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.498785 1997 rpcz_store.cc:276] 0626 08:23:59.229897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.497129 (+1267232us) service_pool.cc:225] Handling call
0626 08:24:00.498330 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.502722 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.552289 2582 raft_consensus.cc:2391] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:00.507167 2001 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:24:00.515321 1925 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.558449 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2075) took 1409 ms. Trace:
I20250626 08:24:00.558905 2000 rpcz_store.cc:276] 0626 08:23:59.138010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.539759 (+1401749us) service_pool.cc:225] Handling call
0626 08:24:00.540131 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.541886 (+ 1755us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.547097 (+ 5211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:00.562537 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.562608 4334 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:00.563550 4334 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:00.564432 4556 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.106s user 0.000s sys 0.009s
I20250626 08:24:00.566329 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.567173 1993 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:00.571364 1997 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:24:00.572315 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2083) took 1281 ms. Trace:
W20250626 08:24:00.572468 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:00.573091 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:00.575891 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.576833 4056 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:00.572793 1997 rpcz_store.cc:276] 0626 08:23:59.305416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.550962 (+1245546us) service_pool.cc:225] Handling call
0626 08:24:00.570665 (+ 19703us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.571900 (+ 1235us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.572195 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.581653 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:00.514081 4546 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20250626 08:24:00.582945 2586 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:00.585976 1928 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:00.579586 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 356) took 1531 ms. Trace:
I20250626 08:24:00.559068 1814 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:00.586823 2002 rpcz_store.cc:276] 0626 08:23:59.021302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.516643 (+1495341us) service_pool.cc:225] Handling call
0626 08:24:00.517243 (+ 600us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.517581 (+ 338us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.517626 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.517718 (+ 92us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:00.517797 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.517899 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.553129 (+ 35230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.585608 2585 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
I20250626 08:24:00.591346 1919 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:00.586685 2583 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:00.588650 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 675) took 2106 ms. Trace:
I20250626 08:24:00.594476 2584 rpcz_store.cc:276] 0626 08:23:58.481987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.391807 (+1909820us) service_pool.cc:225] Handling call
0626 08:24:00.588570 (+196763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.590421 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1927) took 2976 ms. Trace:
I20250626 08:24:00.545094 2579 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:00.504487 4132 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:00.600833 1996 raft_consensus.cc:1273] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:24:00.592610 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:00.477612 2578 rpcz_store.cc:276] 0626 08:23:58.482344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.352682 (+1870338us) service_pool.cc:225] Handling call
0626 08:24:00.353836 (+ 1154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.601634 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2078) took 1435 ms. Trace:
I20250626 08:24:00.584743 3988 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:00.604492 1996 rpcz_store.cc:276] 0626 08:23:59.165789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.545673 (+1379884us) service_pool.cc:225] Handling call
0626 08:24:00.597723 (+ 52050us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.601317 (+ 3594us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.601572 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.602577 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 577) took 3139 ms. Trace:
W20250626 08:24:00.608460 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 115326us
I20250626 08:24:00.608690 2583 rpcz_store.cc:276] 0626 08:23:57.457784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.521234 (+3063450us) service_pool.cc:225] Handling call
0626 08:24:00.521735 (+ 501us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.593383 (+ 71648us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.597164 (+ 3781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.610466 3851 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250626 08:24:00.612975 4488 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.613864 4488 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.615089 2581 raft_consensus.cc:604] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:00.615470 2581 raft_consensus.cc:612] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:00.615760 3973 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:00.616274 3973 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:00.616845 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 674) took 2155 ms. Trace:
I20250626 08:24:00.617230 2582 rpcz_store.cc:276] 0626 08:23:58.461825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.583548 (+2121723us) service_pool.cc:225] Handling call
0626 08:24:00.616204 (+ 32656us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.616484 (+ 280us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.616507 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.616560 (+ 53us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.616602 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.616662 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.616771 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.618297 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45504 (request call id 0) took 2152 ms. Trace:
I20250626 08:24:00.618474 2587 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:00.618726 2581 rpcz_store.cc:276] 0626 08:23:58.541920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.500089 (+1958169us) service_pool.cc:225] Handling call
0626 08:24:00.618078 (+117989us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003c19d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9079443 00007f4ebdfd3c4a 00007f4eb767adc4 00007f4eb767ad52 00007f4eb767ac81 00007f4eb767abfc 00007f4eb7679eb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:00.618196 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261561600}
I20250626 08:24:00.603647 4132 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:00.621150 2587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 567) took 6180 ms. Trace:
I20250626 08:24:00.621845 2585 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.622323 4500 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:00.622812 4500 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:00.630352 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 195575us
I20250626 08:24:00.631750 3988 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:24:00.632340 3851 consensus_queue.cc:1046] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:00.632579 1965 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.633464 3988 raft_consensus.cc:3053] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:00.635524 3988 raft_consensus.cc:738] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:00.636080 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 671) took 3006 ms. Trace:
I20250626 08:24:00.637145 1919 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:00.638705 4056 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:00.633559 3851 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:00.634567 4520 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:00.595894 1923 rpcz_store.cc:276] 0626 08:23:57.607924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.268032 (+2660108us) service_pool.cc:225] Handling call
0626 08:24:00.268510 (+ 478us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.269177 (+ 667us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.269209 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.269861 (+ 652us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.482868 (+213007us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:24:00.482924 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.483041 (+ 117us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:00.584627 (+101586us) raft_consensus.cc:1683] finished
0626 08:24:00.584644 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.584776 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5384,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"thread_start_us":212037,"threads_started":1,"wal-append.queue_time_us":292039}
I20250626 08:24:00.648517 2585 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250626 08:24:00.649660 2585 pending_rounds.cc:85] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting all ops after (but not including) 4
I20250626 08:24:00.641068 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1926) took 3033 ms. Trace:
I20250626 08:24:00.598403 4334 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:00.641865 1925 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:00.643944 1965 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:00.644774 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 474) took 3004 ms. Trace:
I20250626 08:24:00.621496 2587 rpcz_store.cc:276] 0626 08:23:54.441138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:54.441421 (+ 283us) service_pool.cc:225] Handling call
0626 08:23:54.441710 (+ 289us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.618931 (+6177221us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.620931 (+ 2000us) spinlock_profiling.cc:243] Waited 6.18 s on lock 0x7b54003c19d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.621096 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14204009728}
I20250626 08:24:00.654234 1919 rpcz_store.cc:276] 0626 08:23:57.639363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.603655 (+2964292us) service_pool.cc:225] Handling call
0626 08:24:00.636722 (+ 33067us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.643505 (+ 6783us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.643715 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.652554 1814 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:00.650167 2585 pending_rounds.cc:107] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.5
I20250626 08:24:00.665489 2585 raft_consensus.cc:2915] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Aborting config change with OpId 3.5: Aborted: Op aborted by new leader
W20250626 08:24:00.666038 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.663026 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.651991 1927 rpcz_store.cc:276] 0626 08:23:57.607662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.173720 (+2566058us) service_pool.cc:225] Handling call
0626 08:24:00.174258 (+ 538us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.174673 (+ 415us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:00.199908 (+ 25235us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.640736 (+440828us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.640783 (+ 47us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:00.640830 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.640906 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.641014 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":346,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":9451,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":440136,"threads_started":2,"wal-append.queue_time_us":223762}
I20250626 08:24:00.668062 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.666114 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:00.659394 4189 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:00.647927 2580 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:00.676278 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.671828 1920 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:24:00.676857 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.672005 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.651883 4361 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.668668 1998 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:24:00.678578 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2071) took 1655 ms. Trace:
I20250626 08:24:00.664316 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.680191 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:00.682430 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 155643us
W20250626 08:24:00.682821 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.683772 1965 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:00.674113 2585 raft_consensus.cc:2887] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:00.679064 1998 rpcz_store.cc:276] 0626 08:23:59.023142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.649674 (+1626532us) service_pool.cc:225] Handling call
0626 08:24:00.678508 (+ 28834us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.659580 4056 raft_consensus.cc:695] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.689700 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:00.690390 2002 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:00.672432 3851 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:00.642886 2579 rpcz_store.cc:276] 0626 08:23:57.630370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.544126 (+2913756us) service_pool.cc:225] Handling call
0626 08:24:00.544627 (+ 501us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.600860 (+ 56233us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.636005 (+ 35145us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:00.692953 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 83660.8us
W20250626 08:24:00.699657 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.699488 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.701131 1999 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:00.701606 1919 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.706879 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:00.698683 2551 connection.cc:232] RPC call timeout handler was delayed by -2.356258820000221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20250626 08:24:00.710278 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 412011us
W20250626 08:24:00.710583 2551 connection.cc:232] RPC call timeout handler was delayed by -1.9512588200000209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2021s for a response
W20250626 08:24:00.710953 2551 connection.cc:232] RPC call timeout handler was delayed by -1.4145696189998489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
I20250626 08:24:00.709980 1741 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 3 to 4, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 4 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:24:00.711334 2551 connection.cc:232] RPC call timeout handler was delayed by -1.2390696190000199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20250626 08:24:00.712396 3988 consensus_queue.cc:260] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.713986 3988 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.714442 4539 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:00.714680 4545 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:00.715134 4539 raft_consensus.cc:695] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.715814 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:00.716485 1925 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.716121 4539 consensus_queue.cc:237] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.711167 1893 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.710060 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1915) took 3037 ms. Trace:
I20250626 08:24:00.728757 1925 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:00.727865 1927 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
W20250626 08:24:00.717900 1812 leader_election.cc:336] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733): Timed out: RequestConsensusVote RPC to 127.0.171.196:37733 timed out after 1.700s (SENT)
I20250626 08:24:00.701201 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.739008 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:00.739472 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:00.738292 1923 raft_consensus.cc:2391] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:00.741982 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:00.742755 1923 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:00.721596 4333 consensus_queue.cc:1046] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20250626 08:24:00.725368 4167 consensus_meta.cc:287] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.003s
I20250626 08:24:00.748167 1816 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:00.705451 3851 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:00.750593 1920 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.751096 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:00.754588 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.743762 4333 raft_consensus.cc:3053] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:24:00.761675 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 50627.4us
I20250626 08:24:00.729705 1924 rpcz_store.cc:276] 0626 08:23:57.603958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:58.745650 (+1141692us) service_pool.cc:225] Handling call
0626 08:23:58.746082 (+ 432us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:58.746338 (+ 256us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:00.127640 (+1381302us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.598235 (+470595us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.601749 (+ 3514us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.606300 (+ 4551us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.606401 (+ 101us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.641273 (+ 34872us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9184,"prepare.queue_time_us":28579,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2083,"raft.queue_time_us":33072,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":1788959,"threads_started":3,"wal-append.queue_time_us":246596}
I20250626 08:24:00.767241 2550 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:00.716184 4545 consensus_queue.cc:1046] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250626 08:24:00.726544 4503 consensus_queue.cc:1046] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20250626 08:24:00.766651 1893 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:00.764632 4308 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:00.775178 3851 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.806103 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 482) took 2266 ms. Trace:
I20250626 08:24:00.806711 1927 rpcz_store.cc:276] 0626 08:23:58.539603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.761208 (+2221605us) service_pool.cc:225] Handling call
0626 08:24:00.789036 (+ 27828us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.805598 (+ 16562us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.805643 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.805713 (+ 70us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.805769 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.805851 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.806027 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.778112 4500 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.778620 4488 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:00.725484 4056 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.750468 4486 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250626 08:24:00.784200 3973 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.785296 1741 catalog_manager.cc:5582] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 4 to 5, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 5 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
W20250626 08:24:00.821075 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 54527us
I20250626 08:24:00.787670 4496 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:00.791937 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:00.819012 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 61001.7us
I20250626 08:24:00.794874 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.796630 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.826267 1999 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:00.797928 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:00.798353 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:00.793825 2585 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: []
W20250626 08:24:00.799319 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 93043.7us
W20250626 08:24:00.801225 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:00.830364 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.799582 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:00.766577 4538 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:00.760742 4333 raft_consensus.cc:738] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:00.784433 1742 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32866 (request call id 298) took 19335 ms (client timeout 15000 ms). Trace:
W20250626 08:24:00.812624 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 142020us
I20250626 08:24:00.816848 1919 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:00.750929 4167 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.827260 1920 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:00.827430 2001 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.833214 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:00.841847 4562 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
I20250626 08:24:00.848017 3988 raft_consensus.cc:3053] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:00.847898 4333 consensus_queue.cc:260] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.848505 3988 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:00.848421 4439 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:24:00.849431 4333 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.825719 4189 raft_consensus.cc:2762] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250626 08:24:00.846565 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1925) took 3239 ms. Trace:
I20250626 08:24:00.849615 3988 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.828352 4308 raft_consensus.cc:695] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:00.857785 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1936) took 2256 ms. Trace:
I20250626 08:24:00.833288 4488 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:00.835583 4397 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.858225 1927 rpcz_store.cc:276] 0626 08:23:58.601409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.812422 (+2211013us) service_pool.cc:225] Handling call
0626 08:24:00.850748 (+ 38326us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.857442 (+ 6694us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.857469 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.857511 (+ 42us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:00.857543 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.857615 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.857725 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.842574 4062 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:00.863086 1995 raft_consensus.cc:1238] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:24:00.825524 4496 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:00.867097 1998 raft_consensus.cc:2374] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4. Re-sending same reply.
I20250626 08:24:00.864531 1814 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:00.845988 4538 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.850400 4334 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:00.863898 4488 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:00.859133 4545 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:00.864971 1741 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 3, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
I20250626 08:24:00.883138 3988 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.882925 4062 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.885294 2550 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:00.886322 4488 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.891296 4558 consensus_queue.cc:1046] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20250626 08:24:00.891948 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1937) took 1887 ms. Trace:
I20250626 08:24:00.892688 4560 consensus_queue.cc:1046] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20250626 08:24:00.892306 1919 rpcz_store.cc:276] 0626 08:23:59.004077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.856184 (+1852107us) service_pool.cc:225] Handling call
0626 08:24:00.869269 (+ 13085us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.891581 (+ 22312us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.891609 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.891672 (+ 63us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:24:00.891719 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.891779 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.891894 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10309,"raft.queue_time_us":16002,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104}
I20250626 08:24:00.893675 4560 raft_consensus.cc:3053] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:00.886595 4545 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:00.894124 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:00.896164 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 52134us
W20250626 08:24:00.897361 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 131461us
I20250626 08:24:00.872761 4496 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:00.898175 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2092) took 1297 ms. Trace:
I20250626 08:24:00.900266 1995 rpcz_store.cc:276] 0626 08:23:59.592435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.551299 (+958864us) service_pool.cc:225] Handling call
0626 08:24:00.605215 (+ 53916us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.867119 (+261904us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.888360 (+ 21241us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54003a5f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:00.889620 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":579434368}
I20250626 08:24:00.896750 1814 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:24:00.902050 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 44725.8us
I20250626 08:24:00.851843 1926 rpcz_store.cc:276] 0626 08:23:57.607400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:59.278350 (+1670950us) service_pool.cc:225] Handling call
0626 08:23:59.349280 (+ 70930us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:23:59.597149 (+247869us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:23:59.598014 (+ 865us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.811126 (+1213112us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.826776 (+ 15650us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.834569 (+ 7793us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.841387 (+ 6818us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.846493 (+ 5106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":263,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":57816,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":1175394,"threads_started":2,"wal-append.queue_time_us":603498}
I20250626 08:24:00.907735 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1933) took 2302 ms. Trace:
I20250626 08:24:00.908311 1924 rpcz_store.cc:276] 0626 08:23:58.597389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.767497 (+2170108us) service_pool.cc:225] Handling call
0626 08:24:00.828554 (+ 61057us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.886486 (+ 57932us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.886516 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.895198 (+ 8682us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.899175 (+ 3977us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.899239 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.899343 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.869366 4503 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
I20250626 08:24:00.894498 4545 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:00.915095 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 172592us
I20250626 08:24:00.918234 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:00.918649 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:00.874053 4397 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.892496 4558 raft_consensus.cc:3053] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
W20250626 08:24:00.921322 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 62685.3us
I20250626 08:24:00.922173 4262 raft_consensus.cc:3053] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:00.922283 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:00.923426 4062 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:00.895123 4560 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:00.927590 1841 raft_consensus.cc:2391] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:00.927201 2550 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:00.834481 1742 rpcz_store.cc:269] 0626 08:23:41.489003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:41.587344 (+ 98341us) service_pool.cc:225] Handling call
0626 08:24:00.784168 (+19196824us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:23:53.158763 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:23:53.159819 (+ 1056us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:23:53.159928 (+ 109us) write_op.cc:435] Acquired schema lock
0626 08:23:53.159955 (+ 27us) tablet.cc:660] Decoding operations
0626 08:23:53.161410 (+ 1455us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:23:53.161507 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0626 08:23:53.161548 (+ 41us) tablet.cc:683] Acquiring locks for 10 operations
0626 08:23:53.462524 (+300976us) tablet.cc:699] Row locks acquired
0626 08:23:53.467098 (+ 4574us) write_op.cc:260] PREPARE: finished
0626 08:23:53.555955 (+ 88857us) write_op.cc:270] Start()
0626 08:23:53.611956 (+ 56001us) write_op.cc:276] Timestamp: P: 1750926233479656 usec, L: 0
0626 08:23:53.630243 (+ 18287us) op_driver.cc:348] REPLICATION: starting
0626 08:23:53.899078 (+268835us) log.cc:844] Serialized 4069 byte log entry
0626 08:23:57.436139 (+3537061us) op_driver.cc:464] REPLICATION: finished
0626 08:23:58.744229 (+1308090us) write_op.cc:301] APPLY: starting
0626 08:23:58.759675 (+ 15446us) tablet.cc:1365] starting BulkCheckPresence
0626 08:23:58.923910 (+164235us) tablet.cc:1368] finished BulkCheckPresence
0626 08:23:59.022153 (+ 98243us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:23:59.408300 (+386147us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:23:59.632599 (+224299us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0626 08:23:59.641878 (+ 9279us) write_op.cc:312] APPLY: finished
0626 08:23:59.683408 (+ 41530us) log.cc:844] Serialized 77 byte log entry
0626 08:24:00.354218 (+670810us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:00.354581 (+ 363us) write_op.cc:454] Released schema lock
0626 08:24:00.354717 (+ 136us) write_op.cc:341] FINISH: Updating metrics
0626 08:24:00.355369 (+ 652us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:00.355401 (+ 32us) write_op.cc:454] Released schema lock
0626 08:24:00.433767 (+ 78366us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b18000cb1a0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
Metrics: {"mutex_wait_us":13371,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":1254474,"apply.run_cpu_time_us":39719,"apply.run_wall_time_us":1632462,"num_ops":10,"prepare.queue_time_us":894844,"prepare.run_cpu_time_us":49614,"prepare.run_wall_time_us":1016864,"raft.queue_time_us":836084,"raft.run_cpu_time_us":43972,"raft.run_wall_time_us":1386565,"replication_time_us":3677091,"spinlock_wait_cycles":3322240,"thread_start_us":2614877,"threads_started":5,"wal-append.queue_time_us":1047866,"wal-append.run_cpu_time_us":53257,"wal-append.run_wall_time_us":3537454}]]}
I20250626 08:24:00.919895 4439 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:00.931586 1741 catalog_manager.cc:5582] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 4, leader changed from be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:24:00.920648 3886 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:00.926128 4486 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:00.937634 1814 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:00.943765 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:00.951337 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 86948.3us
I20250626 08:24:00.945484 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 483) took 1658 ms. Trace:
I20250626 08:24:00.928678 4262 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:00.951704 1816 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:00.951931 1927 rpcz_store.cc:276] 0626 08:23:59.278949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.869566 (+1590617us) service_pool.cc:225] Handling call
0626 08:24:00.899856 (+ 30290us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.936622 (+ 36766us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.936656 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:00.936702 (+ 46us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.936731 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.936786 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:00.936880 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:00.947008 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:00.931711 3973 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:00.952844 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45623.7us
I20250626 08:24:00.926705 4558 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:00.946615 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:00.954864 4167 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:00.949270 1924 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250626 08:24:00.962666 4560 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:00.946693 4557 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:24:00.966029 4448 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.555s user 0.013s sys 0.000s
I20250626 08:24:00.967350 2001 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:00.964206 4558 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:00.989012 4558 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:00.960381 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:00.987653 4560 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:00.958528 4500 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:00.932231 4463 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20250626 08:24:00.957612 4439 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.977993 2580 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:00.959409 4518 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:00.959522 4486 raft_consensus.cc:695] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:01.012394 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:00.978399 4557 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:00.938053 4308 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.024403 4056 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:01.002043 4109 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
W20250626 08:24:01.032296 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 94716us
I20250626 08:24:01.035283 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:00.994166 4448 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.041947 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 43825.9us
I20250626 08:24:01.046603 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:01.048113 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.049252 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 111896us
I20250626 08:24:01.023934 4557 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:01.055123 1741 catalog_manager.cc:5582] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 1 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:24:01.056349 1923 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:01.044895 3886 raft_consensus.cc:695] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:01.044806 4544 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:01.043180 4109 raft_consensus.cc:695] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:00.998193 4538 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.065542 4518 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:01.066721 4518 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.071949 1841 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:01.071395 1741 catalog_manager.cc:5582] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20250626 08:24:01.075225 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.076258 1841 raft_consensus.cc:2391] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:01.078063 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:01.079645 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.078079 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:01.079336 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 478) took 3438 ms. Trace:
I20250626 08:24:01.082998 1928 rpcz_store.cc:276] 0626 08:23:57.640998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.604472 (+2963474us) service_pool.cc:225] Handling call
0626 08:24:00.637894 (+ 33422us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.638552 (+ 658us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.638582 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.639199 (+ 617us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.919864 (+280665us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.919914 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.919988 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:01.079122 (+159134us) raft_consensus.cc:1683] finished
0626 08:24:01.079147 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.079277 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101499,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164,"thread_start_us":249451,"threads_started":1,"wal-append.queue_time_us":308281}
W20250626 08:24:01.090127 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 145484us
I20250626 08:24:01.090653 1841 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:01.092257 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.103711 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.105193 1846 raft_consensus.cc:2366] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6.
I20250626 08:24:01.102751 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.107225 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.082777 4544 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:01.103557 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.104159 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.077807 2580 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:01.058727 4132 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.118288 2581 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: [4.7-6.8]
I20250626 08:24:01.118992 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:01.046844 4262 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.128154 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.129989 4571 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
I20250626 08:24:01.132388 1890 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:01.104810 4397 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:01.134752 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:01.136953 4109 consensus_queue.cc:237] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:01.141886 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.142870 1847 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:01.146803 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:01.147869 1841 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:24:01.151554 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.152477 1816 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:01.153488 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 93911us
I20250626 08:24:01.151486 2586 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:01.155349 4078 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:01.156657 1812 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:01.158205 1812 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:01.159564 4397 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:01.171962 4568 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20250626 08:24:01.172266 1893 connection.cc:232] RPC call timeout handler was delayed by -2.6631440219998694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20250626 08:24:01.173358 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1939) took 1889 ms. Trace:
I20250626 08:24:01.173856 1919 rpcz_store.cc:276] 0626 08:23:59.283623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.893767 (+1610144us) service_pool.cc:225] Handling call
0626 08:24:00.894192 (+ 425us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.894502 (+ 310us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.894524 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.913000 (+ 18476us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.996411 (+ 83411us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.996463 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.996582 (+ 119us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:01.173065 (+176483us) raft_consensus.cc:1683] finished
0626 08:24:01.173095 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.173284 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":73791,"threads_started":1,"wal-append.queue_time_us":129064}
W20250626 08:24:01.172679 1893 connection.cc:232] RPC call timeout handler was delayed by -2.6469440219998432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
W20250626 08:24:01.175462 1893 connection.cc:232] RPC call timeout handler was delayed by -2.6190440219997981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
W20250626 08:24:01.175812 1893 connection.cc:232] RPC call timeout handler was delayed by -2.5650440219997108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
W20250626 08:24:01.176102 1893 connection.cc:232] RPC call timeout handler was delayed by -2.4903440220000448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20250626 08:24:01.176411 1893 connection.cc:232] RPC call timeout handler was delayed by -2.285144021999713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
W20250626 08:24:01.176725 1893 connection.cc:232] RPC call timeout handler was delayed by -2.1546440219999567s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20250626 08:24:01.177230 1893 connection.cc:232] RPC call timeout handler was delayed by -2.1168440219998956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
W20250626 08:24:01.177532 1893 connection.cc:232] RPC call timeout handler was delayed by -2.1060440219998782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
W20250626 08:24:01.177876 1893 connection.cc:232] RPC call timeout handler was delayed by -2.0961440219998622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20250626 08:24:01.177994 4062 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:01.178409 1893 connection.cc:232] RPC call timeout handler was delayed by -2.0781440219998331s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
I20250626 08:24:01.178727 4062 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:01.178781 1893 connection.cc:232] RPC call timeout handler was delayed by -2.0466440219997821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
W20250626 08:24:01.182101 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9147044490000553s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1545s for a response
W20250626 08:24:01.182435 1893 connection.cc:232] RPC call timeout handler was delayed by -1.7005044490001637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
I20250626 08:24:01.181478 4062 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:01.184553 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.186720 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:01.192970 1928 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:01.199384 2001 raft_consensus.cc:2391] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:01.177676 4519 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:01.203871 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 44539.1us
W20250626 08:24:01.196689 4563 log.cc:927] Time spent T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20250626 08:24:01.213145 1926 raft_consensus.cc:1238] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:01.208027 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.213814 4569 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250626 08:24:01.214288 4573 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:01.217090 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 679) took 2733 ms. Trace:
I20250626 08:24:01.197877 3886 consensus_queue.cc:237] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.218997 4544 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.220813 4576 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:01.222209 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.228449 4581 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:24:01.228477 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.242208 4519 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:01.235023 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:01.130756 4486 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.240031 4573 raft_consensus.cc:695] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:01.248725 1928 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250626 08:24:01.202418 4448 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:01.116552 4439 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.245550 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1938) took 2241 ms. Trace:
I20250626 08:24:01.274824 1920 rpcz_store.cc:276] 0626 08:23:59.004387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.847378 (+1842991us) service_pool.cc:225] Handling call
0626 08:24:00.847765 (+ 387us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:00.876392 (+ 28627us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:24:00.883290 (+ 6898us) log.cc:844] Serialized 14 byte log entry
0626 08:24:01.000808 (+117518us) log.cc:844] Serialized 14 byte log entry
0626 08:24:01.214835 (+214027us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b18002f3ce0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:01.244874 (+ 30039us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18002f3ce0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb609825a 00007f4ebf55a229 00007f4eb5f599c0 00007f4eb611d5e4 00007f4eb611d555 00007f4eb9076c50 00007f4eb90892f6 00007f4eb9099e67 00007f4eb9099de8
0626 08:24:01.245074 (+ 200us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:01.245141 (+ 67us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:01.245203 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:01.245298 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.245477 (+ 179us) inbound_call.cc:173] Queueing success response
0626 08:24:01.248108 (+ 2631us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b18002f3ce0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380
Metrics: {"mutex_wait_us":260337,"prepare.queue_time_us":217106,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":274048,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":8492,"spinlock_wait_cycles":21042048,"thread_start_us":116856,"threads_started":1,"wal-append.queue_time_us":169553}
I20250626 08:24:01.276505 1995 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:01.277088 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:01.253278 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.249140 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.229765 2586 rpcz_store.cc:276] 0626 08:23:58.483317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.592153 (+2108836us) service_pool.cc:225] Handling call
0626 08:24:00.643059 (+ 50906us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:01.154550 (+511491us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:01.216798 (+ 62248us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b540066eed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:01.216996 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1168809344}
W20250626 08:24:01.252318 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 120047us
W20250626 08:24:01.262153 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 45082.5us
W20250626 08:24:01.280081 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.262614 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 96153.6us
I20250626 08:24:01.194466 4397 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.279023 1812 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:01.263116 4558 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:01.281926 1926 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:01.284965 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.285596 2001 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:01.266826 4576 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:01.297525 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.335302 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.324438 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.342823 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:01.290237 4356 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.851s user 0.010s sys 0.003s
I20250626 08:24:01.346478 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1931) took 3707 ms. Trace:
I20250626 08:24:01.346796 1921 rpcz_store.cc:276] 0626 08:23:57.638683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.561020 (+2922337us) service_pool.cc:225] Handling call
0626 08:24:00.614188 (+ 53168us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.614763 (+ 575us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.614807 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.615314 (+ 507us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.898756 (+283442us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:00.898791 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.898851 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:01.346285 (+447434us) raft_consensus.cc:1683] finished
0626 08:24:01.346307 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.346425 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":43193,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":12546,"thread_start_us":283083,"threads_started":1,"wal-append.queue_time_us":324789}
I20250626 08:24:01.345256 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.348157 2001 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:01.253681 4496 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.345507 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:01.282269 4581 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:01.350025 1920 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 3: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:01.352064 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:01.354769 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 228345us
I20250626 08:24:01.336263 4576 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:01.328270 4109 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:01.357581 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:01.358506 1927 raft_consensus.cc:2408] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:24:01.335479 4558 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:01.360358 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.362435 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.363022 2001 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:01.365103 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.365952 2001 raft_consensus.cc:2366] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:01.286619 4334 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20250626 08:24:01.363129 2002 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:01.368794 1920 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20250626 08:24:01.326766 4488 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.327900 4056 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20250626 08:24:01.325451 4520 consensus_queue.cc:1046] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20250626 08:24:01.341248 4078 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20250626 08:24:01.375746 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.376967 1928 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:01.373296 1926 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:01.361104 4132 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:01.397725 1920 raft_consensus.cc:1238] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:01.372437 4519 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:24:01.408118 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.412465 2001 raft_consensus.cc:2391] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:01.417479 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 481) took 2904 ms. Trace:
I20250626 08:24:01.418581 1923 rpcz_store.cc:276] 0626 08:23:58.512992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.766079 (+2253087us) service_pool.cc:225] Handling call
0626 08:24:00.826349 (+ 60270us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:01.056877 (+230528us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:01.417162 (+360285us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54002dfad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:01.417399 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":527259520}
I20250626 08:24:01.422840 3851 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:01.442770 4056 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:01.443163 3988 consensus_meta.cc:287] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.548s user 0.011s sys 0.000s
I20250626 08:24:01.443398 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.443915 4585 raft_consensus.cc:3053] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:24:01.441289 3851 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.410022 4519 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:01.383368 4109 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.467244 4546 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20250626 08:24:01.453574 1926 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20250626 08:24:01.509325 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.500309 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.529551 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.495312 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.530169 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 252448us
I20250626 08:24:01.534215 2001 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
W20250626 08:24:01.539265 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 67571.6us
W20250626 08:24:01.497757 4579 log.cc:927] Time spent T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
W20250626 08:24:01.487730 4575 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.205s user 0.001s sys 0.000s
W20250626 08:24:01.548228 4495 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.016s sys 0.024s
W20250626 08:24:01.549248 4539 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.030s sys 0.056s
I20250626 08:24:01.430501 4078 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.105s
W20250626 08:24:01.552579 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 175922us
I20250626 08:24:01.530102 4519 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:01.551998 2002 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:01.559249 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.512677 4585 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:01.558636 4539 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.030s sys 0.056s
W20250626 08:24:01.576277 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 233510us
W20250626 08:24:01.557638 4545 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.627s user 0.005s sys 0.007s
W20250626 08:24:01.583818 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.556309 4495 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.787s user 0.016s sys 0.024s
I20250626 08:24:01.562048 4397 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:01.615413 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.621253 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.621831 3851 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:01.622625 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.503033 4558 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.638810 1995 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:01.643378 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1950) took 1167 ms. Trace:
I20250626 08:24:01.648502 1924 rpcz_store.cc:276] 0626 08:24:00.475694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.977496 (+501802us) service_pool.cc:225] Handling call
0626 08:24:00.987447 (+ 9951us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.987700 (+ 253us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.987730 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.988261 (+ 531us) log.cc:844] Serialized 26 byte log entry
0626 08:24:01.044429 (+ 56168us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:01.044463 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:01.044547 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:01.643065 (+598518us) raft_consensus.cc:1683] finished
0626 08:24:01.643094 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.643301 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":55697,"threads_started":1,"wal-append.queue_time_us":205425}
I20250626 08:24:01.619581 4496 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:01.468597 4581 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.652092 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 71139.3us
I20250626 08:24:01.653100 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.573004 4132 consensus_queue.cc:1046] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20250626 08:24:01.660334 4518 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:01.660840 4518 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:01.662093 4518 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.665735 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.643517 4560 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:01.467770 4573 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.657528 4333 consensus_meta.cc:287] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.806s user 0.012s sys 0.002s
I20250626 08:24:01.664921 4488 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:01.673514 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39250 (request call id 1091) took 1438 ms. Trace:
W20250626 08:24:01.677366 4564 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.746s user 0.001s sys 0.002s
I20250626 08:24:01.617707 4545 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:01.687811 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 215818us
I20250626 08:24:01.688951 1920 rpcz_store.cc:276] 0626 08:24:00.220433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.516323 (+1295890us) service_pool.cc:225] Handling call
0626 08:24:01.653331 (+137008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:01.696040 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.697144 1995 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:01.697896 1928 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:01.698908 1921 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20250626 08:24:01.699507 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 74867.7us
W20250626 08:24:01.709621 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 237601us
I20250626 08:24:01.699146 2551 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:01.704334 1893 leader_election.cc:336] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293): Timed out: RequestConsensusVote RPC to 127.0.171.195:38293 timed out after 1.544s (SENT)
I20250626 08:24:01.673547 4560 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.716845 1893 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:01.717813 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.721298 1995 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250626 08:24:01.722532 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.723537 2001 raft_consensus.cc:2366] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
W20250626 08:24:01.724233 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 86568.2us
I20250626 08:24:01.730880 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.731742 1928 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:01.701223 2579 raft_consensus.cc:1238] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:01.732290 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:01.734647 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.738492 2002 raft_consensus.cc:2366] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
W20250626 08:24:01.738632 4594 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
I20250626 08:24:01.739415 4334 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:24:01.741719 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:01.743294 4056 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:01.743819 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 49015.9us
I20250626 08:24:01.744693 1816 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:01.745954 4598 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:01.746410 4598 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:01.746591 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:01.747258 4598 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:01.748739 4598 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:01.749531 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:01.750032 4561 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:01.750483 4561 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:01.751549 4056 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:01.751340 4561 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.752394 4561 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:01.752257 4056 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:01.752240 1816 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:01.755173 4599 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:01.756014 4599 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:01.756459 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:01.756786 4599 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:01.762399 4599 raft_consensus.cc:695] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:01.763583 4599 consensus_queue.cc:237] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:01.764645 4518 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:01.722081 4544 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:01.737807 4601 log.cc:927] Time spent T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20250626 08:24:01.783318 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:01.786339 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.785486 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:01.793782 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.854360 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:01.867334 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 172358us
I20250626 08:24:01.880825 2582 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250626 08:24:01.882449 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 147502us
I20250626 08:24:01.776440 4109 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:01.903534 4496 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:01.918350 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.945875 1999 raft_consensus.cc:2366] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:24:01.919340 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.919657 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.948140 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2082) took 2710 ms. Trace:
I20250626 08:24:01.948411 2000 raft_consensus.cc:2366] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:24:01.919965 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.923046 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:01.918290 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.952886 1995 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:01.949604 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.954233 1924 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:01.939013 1926 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:01.918629 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.956682 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.926972 2586 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:01.958577 1965 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:01.961208 1928 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:01.942044 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:01.917531 4262 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:01.966049 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.978763 2002 raft_consensus.cc:2366] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:01.919051 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:01.986114 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 44218.3us
I20250626 08:24:01.987143 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.989956 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:01.966636 4495 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:01.917994 3851 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20250626 08:24:01.987982 1993 raft_consensus.cc:2374] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6. Re-sending same reply.
I20250626 08:24:01.968127 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:01.968967 1996 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:24:01.969440 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:01.970119 2001 raft_consensus.cc:2366] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:01.923120 4545 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:01.972338 4560 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:01.963193 4601 log.cc:927] Time spent T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250626 08:24:01.909042 4613 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:24:01.983124 4605 log.cc:927] Time spent T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20250626 08:24:01.984809 4539 consensus_queue.cc:1046] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250626 08:24:01.948657 1994 rpcz_store.cc:276] 0626 08:23:59.237572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.559349 (+1321777us) service_pool.cc:225] Handling call
0626 08:24:00.618442 (+ 59093us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.618899 (+ 457us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.618926 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.619466 (+ 540us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.854268 (+234802us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:24:00.854312 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.854379 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:01.676560 (+822181us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:01.942450 (+265890us) raft_consensus.cc:1683] finished
0626 08:24:01.942478 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:01.948057 (+ 5579us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1198,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"spinlock_wait_cycles":28138368,"thread_start_us":234422,"threads_started":1,"wal-append.queue_time_us":305906}
I20250626 08:24:02.015944 4495 raft_consensus.cc:695] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:02.010803 1928 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:01.967953 1998 raft_consensus.cc:2408] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250626 08:24:01.921337 4056 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:02.012598 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:02.143924 1924 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:02.016846 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:02.145205 1995 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:02.017138 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.976740 4109 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:02.022018 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.024089 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:02.147979 1926 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:02.148576 2000 raft_consensus.cc:2374] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5. Re-sending same reply.
I20250626 08:24:02.149032 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:02.025168 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:02.149880 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:02.150844 2000 raft_consensus.cc:2366] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:02.151147 2002 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20250626 08:24:02.013731 4557 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.934s user 0.009s sys 0.004s
I20250626 08:24:02.046541 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:01.911584 4496 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:02.048482 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:02.020022 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 130968us
I20250626 08:24:02.006778 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:02.061894 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:02.073634 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:02.036044 4486 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:02.132350 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:02.270789 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:02.271685 1994 raft_consensus.cc:2408] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250626 08:24:02.132898 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 97826.9us
I20250626 08:24:02.134687 3851 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.135147 4333 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:02.136729 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:02.139318 4576 consensus_meta.cc:287] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.773s user 0.012s sys 0.000s
I20250626 08:24:02.156131 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:02.139256 3988 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.266688 4557 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.284636 3851 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.284904 3988 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.286325 4557 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.286613 3851 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.287004 2582 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:02.287227 1919 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250626 08:24:02.272538 2550 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:02.288034 3851 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:02.288511 3851 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.289594 4609 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:02.292060 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:02.286944 3988 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.292925 3988 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.293385 3988 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:02.294528 4606 log.cc:927] Time spent T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20250626 08:24:02.295234 4189 consensus_queue.cc:1046] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:02.295909 3988 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.295970 4109 consensus_queue.cc:1046] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250626 08:24:02.296746 4557 raft_consensus.cc:3053] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:02.297200 4557 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:02.292352 4609 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:02.274403 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.298655 4609 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.266968 2001 raft_consensus.cc:2374] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5. Re-sending same reply.
I20250626 08:24:02.290482 4603 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250626 08:24:02.311738 4555 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.398s user 0.000s sys 0.017s
I20250626 08:24:02.313521 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1928) took 4691 ms. Trace:
I20250626 08:24:02.313856 1918 rpcz_store.cc:276] 0626 08:23:57.622208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.296006 (+2673798us) service_pool.cc:225] Handling call
0626 08:24:00.296476 (+ 470us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.300058 (+ 3582us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.300102 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.300731 (+ 629us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.699377 (+398646us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:00.724505 (+ 25128us) log.cc:844] Serialized 14 byte log entry
0626 08:24:00.724712 (+ 207us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.724793 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:02.311034 (+1586241us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:02.313134 (+ 2100us) raft_consensus.cc:1683] finished
0626 08:24:02.313276 (+ 142us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:02.313450 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2944,"prepare.queue_time_us":26502,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":129988,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":272,"spinlock_wait_cycles":54661376,"thread_start_us":398245,"threads_started":1,"wal-append.queue_time_us":553918}
I20250626 08:24:02.318374 2582 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250626 08:24:02.318634 3851 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.271728 3886 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:24:02.306071 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:02.329275 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 101463us
I20250626 08:24:02.227159 4598 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.093s
W20250626 08:24:02.323217 4500 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.009s sys 0.007s
W20250626 08:24:02.334735 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.011s sys 0.012s
W20250626 08:24:02.339227 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.011s sys 0.012s
I20250626 08:24:02.274732 4056 consensus_queue.cc:1035] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250626 08:24:02.327996 1993 raft_consensus.cc:2366] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
W20250626 08:24:02.363201 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.008s sys 0.019s
I20250626 08:24:02.325276 3886 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:02.364220 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.008s sys 0.019s
I20250626 08:24:02.302242 4333 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:02.339056 4500 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s user 0.009s sys 0.008s
W20250626 08:24:02.382249 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 52033.6us
I20250626 08:24:02.328382 4189 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.037s
I20250626 08:24:02.305881 4576 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:02.385529 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 397425us
W20250626 08:24:02.382800 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 752533us
W20250626 08:24:02.374940 1893 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.010s sys 0.012s
W20250626 08:24:02.332533 4538 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.007s sys 0.009s
W20250626 08:24:02.387411 1893 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.643s user 0.010s sys 0.013s
W20250626 08:24:02.388684 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 408477us
W20250626 08:24:02.368234 4062 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.005s sys 0.015s
W20250626 08:24:02.391592 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 647114us
I20250626 08:24:02.367686 3886 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:02.318411 4495 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.366182 4613 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:02.383203 4578 raft_consensus.cc:3053] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Stepping down as leader of term 6
W20250626 08:24:02.383373 4546 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.003s sys 0.019s
I20250626 08:24:02.397727 1816 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:02.398243 2582 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:02.431875 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 44742.3us
W20250626 08:24:02.418709 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:02.396271 4062 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s user 0.006s sys 0.017s
W20250626 08:24:02.393641 4538 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s user 0.013s sys 0.009s
I20250626 08:24:02.398870 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:02.433563 4546 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.003s sys 0.020s
W20250626 08:24:02.494185 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 44680us
I20250626 08:24:02.430790 4578 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:02.507401 4616 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.208s
I20250626 08:24:02.515640 1967 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:02.366492 4557 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.460844 4560 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.535900 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 589) took 1479 ms. Trace:
I20250626 08:24:02.559544 2581 rpcz_store.cc:276] 0626 08:24:01.042378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.042523 (+ 145us) service_pool.cc:225] Handling call
0626 08:24:01.117880 (+ 75357us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:01.118858 (+ 978us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:02.521694 (+1402836us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b540050f350. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:02.521868 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":159589120}
W20250626 08:24:02.537416 4519 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.957s user 0.009s sys 0.010s
I20250626 08:24:02.521076 3988 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.545069 4624 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:02.538302 4496 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:02.537777 4633 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:02.550180 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 159508us
W20250626 08:24:02.550884 4567 log.cc:927] Time spent T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.615s user 0.017s sys 0.013s
I20250626 08:24:02.537001 4570 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.552392 4333 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:02.553613 4262 consensus_meta.cc:287] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.572s user 0.008s sys 0.011s
W20250626 08:24:02.578994 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.558738 2580 raft_consensus.cc:2408] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250626 08:24:02.584267 2583 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:02.575392 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:02.587296 1965 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.562529 4062 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:02.587152 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 249911us
I20250626 08:24:02.502358 4613 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.590220 4633 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:02.596657 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:02.578508 4557 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:02.597455 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:02.629115 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 113592us
I20250626 08:24:02.630877 2581 raft_consensus.cc:2391] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:02.600348 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.600961 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.603066 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:02.629228 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 49093.7us
I20250626 08:24:02.631956 1843 raft_consensus.cc:2408] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250626 08:24:02.604453 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.607748 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 479) took 4941 ms. Trace:
I20250626 08:24:02.610145 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.610877 1847 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:02.632668 1925 rpcz_store.cc:276] 0626 08:23:57.666362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.745484 (+3079122us) service_pool.cc:225] Handling call
0626 08:24:00.766720 (+ 21236us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:00.770647 (+ 3927us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:00.770675 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:00.771163 (+ 488us) log.cc:844] Serialized 26 byte log entry
0626 08:24:00.928601 (+157438us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:00.928795 (+ 194us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:00.928876 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:02.482825 (+1553949us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:02.607388 (+124563us) raft_consensus.cc:1683] finished
0626 08:24:02.607410 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:02.607684 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3462016,"thread_start_us":157046,"threads_started":1,"wal-append.queue_time_us":163979}
I20250626 08:24:02.610798 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.611598 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.613395 1849 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:02.579880 3988 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:02.635037 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:02.618774 4638 consensus_queue.cc:1046] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:02.620570 4616 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:02.620757 4637 consensus_queue.cc:1046] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:02.582557 4570 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:02.591655 2585 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:02.653024 1848 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:02.618801 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1824) took 1428 ms. Trace:
I20250626 08:24:02.653533 1921 rpcz_store.cc:276] 0626 08:24:01.190385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.352272 (+161887us) service_pool.cc:225] Handling call
0626 08:24:01.698584 (+346312us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:01.699374 (+ 790us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:02.618591 (+919217us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400367ed0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:02.618745 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":762561280}
I20250626 08:24:02.596794 4599 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:02.631199 1967 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:24:02.636545 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 50533.8us
I20250626 08:24:02.639986 2586 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:02.640172 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.640177 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.656469 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:02.641361 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:02.641582 1842 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:02.640542 4573 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20250626 08:24:02.647060 1845 raft_consensus.cc:2408] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:24:02.646973 4361 consensus_queue.cc:1046] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:02.647779 1850 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:02.659433 1847 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:02.577275 4519 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.648352 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.648927 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.651279 4638 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:02.651744 4617 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.285s user 0.000s sys 0.001s
I20250626 08:24:02.565521 4560 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:02.611629 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.668999 1846 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250626 08:24:02.657712 1849 raft_consensus.cc:2391] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:02.652103 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:02.592871 4624 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:02.628185 4333 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:24:02.660651 1844 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:02.665191 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.703179 1843 raft_consensus.cc:2408] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:24:02.551484 4578 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.694730 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.695588 3988 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:02.696518 4519 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.658305 4599 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:02.660182 4576 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:02.698346 4633 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:02.667528 4638 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:02.700337 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.653373 4614 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:02.716542 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 320548us
W20250626 08:24:02.698040 4561 consensus_meta.cc:287] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.945s user 0.014s sys 0.000s
I20250626 08:24:02.629089 4062 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.715385 1850 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:02.750936 1845 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:02.692231 4486 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:02.701959 4333 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:02.770434 1920 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20250626 08:24:02.772022 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 140893us
I20250626 08:24:02.755686 1814 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:02.776620 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:02.766886 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.711220 4520 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:02.772629 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 53557.2us
I20250626 08:24:02.785858 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:02.785791 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 196941us
I20250626 08:24:02.786664 1998 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:02.748356 4496 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:02.745893 4558 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:02.789453 2002 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:02.746469 4334 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:02.754938 3988 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:02.794083 1995 pending_rounds.cc:85] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 4
I20250626 08:24:02.756395 4614 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.769815 4486 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:02.798945 1967 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:02.780074 4333 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:02.796761 1994 raft_consensus.cc:3053] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:02.803224 1994 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:02.753099 4189 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:02.796032 1995 pending_rounds.cc:107] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250626 08:24:02.794492 1814 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:02.804342 1995 raft_consensus.cc:2887] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:02.748998 4573 consensus_queue.cc:1230] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250626 08:24:02.776067 4361 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:02.788934 4576 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:02.783069 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 587) took 1856 ms. Trace:
I20250626 08:24:02.791304 1993 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250626 08:24:02.797767 3988 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:02.805902 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 164601us
W20250626 08:24:02.830283 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.807833 4486 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:02.846135 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 47296.5us
I20250626 08:24:02.886375 1967 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:02.710697 4624 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:02.882320 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:02.789820 4496 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:02.840996 1994 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.892035 1994 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:02.892669 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 290165us
W20250626 08:24:02.888437 1924 consensus_meta.cc:287] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.744s user 0.010s sys 0.003s
I20250626 08:24:02.889099 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.897074 1924 raft_consensus.cc:2408] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250626 08:24:02.902272 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 115067us
I20250626 08:24:02.791370 4520 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.799340 4558 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.798499 4629 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20250626 08:24:02.951031 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.887317 3988 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:02.915414 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:02.919354 1893 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:02.906687 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:02.943814 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.953094 1847 raft_consensus.cc:2391] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:02.944231 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.945835 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.946056 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:02.879439 4576 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:02.886286 4538 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:02.963043 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.845737 4634 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250626 08:24:02.839262 4334 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.960521 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:02.950277 4619 log.cc:927] Time spent T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20250626 08:24:02.965988 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:02.960810 1846 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:02.845942 4262 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.103s
I20250626 08:24:02.874593 4189 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:02.985996 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 181188us
I20250626 08:24:02.988090 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:03.002243 1847 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:02.961300 4576 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:03.019325 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 217960us
I20250626 08:24:03.020409 2551 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:03.021809 4609 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:03.022310 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:03.024129 4609 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:03.025684 2551 leader_election.cc:400] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6.
I20250626 08:24:03.026980 2551 leader_election.cc:403] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:03.028203 4609 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6.
I20250626 08:24:03.029855 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:02.883895 4599 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:02.879053 2579 rpcz_store.cc:276] 0626 08:24:00.926200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:00.929182 (+ 2982us) service_pool.cc:225] Handling call
0626 08:24:00.959831 (+ 30649us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:01.737077 (+777246us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:02.751286 (+1014209us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54006149d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:02.779679 (+ 28393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616148352}
I20250626 08:24:02.997630 4570 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:02.994707 4538 raft_consensus.cc:695] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:03.066256 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 179970us
I20250626 08:24:03.074415 2551 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:03.077836 4620 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250626 08:24:03.079221 4189 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.079887 4189 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:03.082154 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 179151us
I20250626 08:24:03.082944 1893 leader_election.cc:400] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:24:03.083115 4334 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.083472 4520 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.086380 4062 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.089581 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:03.090147 1998 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:03.090353 1997 raft_consensus.cc:2366] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:03.090883 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1708) took 1174 ms. Trace:
I20250626 08:24:03.091219 1997 rpcz_store.cc:276] 0626 08:24:01.916765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.918443 (+ 1678us) service_pool.cc:225] Handling call
0626 08:24:03.089429 (+1170986us) spinlock_profiling.cc:243] Waited 1.68 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:03.090809 (+ 1380us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3874560}
I20250626 08:24:03.083240 1893 leader_election.cc:403] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:03.095760 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:03.098488 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:03.037874 4609 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:24:03.110329 1924 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:24:03.111037 1924 raft_consensus.cc:604] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:03.111349 1924 raft_consensus.cc:612] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:03.115483 1842 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:03.116770 1848 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
W20250626 08:24:03.118487 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 233205us
I20250626 08:24:03.119477 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:03.123049 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.120153 2582 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:03.127687 1968 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:03.177795 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.178864 1890 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:03.179514 1926 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250626 08:24:03.181648 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 107351us
I20250626 08:24:03.185359 4334 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:03.185938 2551 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:03.185961 4334 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:02.985380 4496 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.100677 4599 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:03.186666 4334 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:03.187387 4496 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:03.194327 4334 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.194777 1849 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:03.122339 4520 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.121277 4657 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:03.230480 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 153319us
I20250626 08:24:03.126840 4439 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.131968 4638 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:03.202454 4546 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250626 08:24:03.231539 1967 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:03.232185 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.203538 4609 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.219231 4545 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEARNER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:03.233853 4664 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
W20250626 08:24:03.233979 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 48155.6us
W20250626 08:24:03.476473 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 393614us
I20250626 08:24:03.476715 2551 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:03.234452 4663 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:03.234813 4642 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250626 08:24:03.235080 4649 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:03.235461 4650 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
W20250626 08:24:03.374053 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.433051 1847 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:03.198918 4624 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:03.475062 1925 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.006s sys 0.015s
I20250626 08:24:03.477443 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:03.204407 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:03.226145 4062 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:03.475489 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 383292us
W20250626 08:24:03.500334 1925 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.541s user 0.006s sys 0.015s
W20250626 08:24:03.501245 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 269791us
W20250626 08:24:03.503631 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:03.543740 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.504704 1924 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:03.510219 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.547874 2586 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:03.481855 4631 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.094s sys 0.107s
W20250626 08:24:03.564298 4631 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.095s sys 0.108s
I20250626 08:24:03.565567 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:03.517328 4634 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.003s sys 0.020s
W20250626 08:24:03.568581 4634 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.003s sys 0.021s
I20250626 08:24:03.228111 4520 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:03.577553 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 77307.3us
I20250626 08:24:03.469511 4519 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:03.585855 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:03.587302 1994 consensus_meta.cc:287] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.695s user 0.009s sys 0.000s
I20250626 08:24:03.587989 1994 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250626 08:24:03.588299 1994 pending_rounds.cc:85] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 5
I20250626 08:24:03.588521 1994 pending_rounds.cc:107] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250626 08:24:03.588754 1994 raft_consensus.cc:2887] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250626 08:24:03.590938 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 355035us
I20250626 08:24:03.502441 4649 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:03.501928 4496 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:24:03.500684 4495 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.596138 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:03.538296 4581 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:03.608130 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:03.598845 4519 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.609686 2586 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 7: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:03.598856 4649 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:03.611364 4519 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:03.613869 3988 consensus_meta.cc:287] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.629s user 0.009s sys 0.000s
I20250626 08:24:03.614375 3988 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.614248 4495 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:03.619292 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:03.623649 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.438310 4439 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.651940 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:03.652740 2582 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:03.475852 4609 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.630911 4662 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.636611 4334 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.608002 4634 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:03.581583 4062 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.641115 4496 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:03.643371 2579 raft_consensus.cc:2391] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:03.618196 4558 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.624128 1967 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:03.686916 4634 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:03.688561 4062 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.467314 4538 consensus_queue.cc:237] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.701078 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.701366 1997 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:03.701490 1998 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:03.701900 2002 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:03.702340 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.702517 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.702844 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.703411 2002 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:03.703763 1993 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250626 08:24:03.704722 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:03.690110 4646 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.615s user 0.004s sys 0.000s
I20250626 08:24:03.706787 4397 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:24:03.707293 4665 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.707839 4397 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.707902 4665 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.625479 4663 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:03.708285 4585 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.709194 4496 raft_consensus.cc:3053] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Stepping down as leader of term 5
W20250626 08:24:03.709647 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 85640.8us
I20250626 08:24:03.708397 4397 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.701225 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:03.711594 1967 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:03.712551 1998 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:03.709606 4648 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.089s user 0.000s sys 0.003s
I20250626 08:24:03.730991 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:03.715770 2001 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:03.709558 4496 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:03.742588 2001 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250626 08:24:03.714970 1997 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:03.698727 4634 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:03.720939 1993 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20250626 08:24:03.718782 4062 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:03.755034 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.757517 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:03.766393 2586 raft_consensus.cc:3053] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:03.766785 2586 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:03.766912 1994 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:24:03.767416 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.012s sys 0.030s
W20250626 08:24:03.768043 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 640555us
W20250626 08:24:03.768819 1999 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.532s user 0.018s sys 0.025s
I20250626 08:24:03.770098 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 378) took 1009 ms. Trace:
I20250626 08:24:03.730262 4560 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:03.765156 2582 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:03.771663 1999 rpcz_store.cc:276] 0626 08:24:02.760661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.760817 (+ 156us) service_pool.cc:225] Handling call
0626 08:24:02.782359 (+ 21542us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:02.788881 (+ 6522us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:03.023601 (+234720us) log.cc:844] Serialized 14 byte log entry
0626 08:24:03.769614 (+746013us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:03.769685 (+ 71us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:03.769751 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:03.769842 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:03.770030 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123731,"prepare.queue_time_us":297153,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":430448,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":17561,"thread_start_us":852366,"threads_started":3,"wal-append.queue_time_us":442533}
I20250626 08:24:03.787921 1890 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:03.768934 1968 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:03.736286 4558 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:03.774444 4062 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:03.789338 4539 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20250626 08:24:03.791529 4334 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:03.699075 4662 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.791635 2586 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.765537 1993 pending_rounds.cc:85] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 4
I20250626 08:24:03.792722 1993 pending_rounds.cc:107] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.5
W20250626 08:24:03.756413 4573 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.009s sys 0.030s
I20250626 08:24:03.786451 4663 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250626 08:24:03.793156 4573 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.009s sys 0.033s
I20250626 08:24:03.793718 1993 raft_consensus.cc:2915] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Aborting config change with OpId 3.5: Aborted: Op aborted by new leader
W20250626 08:24:03.755859 4651 log.cc:927] Time spent T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.490s user 0.000s sys 0.003s
W20250626 08:24:03.740819 4595 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.007s sys 0.030s
I20250626 08:24:03.806617 1994 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250626 08:24:03.806543 1998 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:03.749195 4520 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:03.784570 4361 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.824592 2002 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:03.827260 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:03.794175 4500 raft_consensus.cc:3053] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:03.829491 4500 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:03.829890 4397 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.829634 4496 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.831007 4496 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:03.830649 4500 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.831977 4500 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:03.833307 4495 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:03.807040 1993 raft_consensus.cc:2887] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20250626 08:24:03.834645 4669 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20250626 08:24:03.835757 4680 consensus_queue.cc:1046] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20250626 08:24:03.798189 4599 raft_consensus.cc:3053] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:03.845975 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:03.811944 4595 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.007s sys 0.031s
I20250626 08:24:03.791256 4062 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:03.825172 4539 consensus_queue.cc:1230] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250626 08:24:03.840425 4361 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:03.848259 4662 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:03.865700 4361 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:03.871656 4439 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:03.872478 4681 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:03.875489 4062 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:03.878661 4672 log.cc:927] Time spent T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20250626 08:24:03.879483 1926 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:24:03.880662 4667 raft_consensus.cc:3053] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:03.880826 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.881105 4667 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:03.881443 1926 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:03.882272 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.813050 3988 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:03.882696 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:03.851449 2582 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:03.852155 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:03.854636 4656 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
W20250626 08:24:03.849483 4189 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.769s user 0.010s sys 0.000s
I20250626 08:24:03.858754 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 505) took 1536 ms. Trace:
I20250626 08:24:03.899327 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 509) took 1020 ms. Trace:
I20250626 08:24:03.899739 1918 rpcz_store.cc:276] 0626 08:24:02.322561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.325259 (+ 2698us) service_pool.cc:225] Handling call
0626 08:24:02.334792 (+ 9533us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:02.373639 (+ 38847us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:02.373675 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:02.397102 (+ 23427us) log.cc:844] Serialized 26 byte log entry
0626 08:24:02.893192 (+496090us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:02.893249 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:02.893339 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:03.858512 (+965173us) raft_consensus.cc:1683] finished
0626 08:24:03.858538 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:03.858692 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8554,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":16176,"thread_start_us":387192,"threads_started":1,"wal-append.queue_time_us":599999}
I20250626 08:24:03.902535 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.881893 4667 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:03.858326 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 380935us
I20250626 08:24:03.812708 4573 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:03.906983 4667 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:03.906816 4573 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.810665 4334 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:03.911211 1994 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:24:03.908370 1893 leader_election.cc:400] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:03.911918 1893 leader_election.cc:403] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250626 08:24:03.908905 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.912689 1918 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
W20250626 08:24:03.912789 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 327625us
I20250626 08:24:03.914407 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:03.914517 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.899950 1925 rpcz_store.cc:276] 0626 08:24:02.879310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.879516 (+ 206us) service_pool.cc:225] Handling call
0626 08:24:02.885756 (+ 6240us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:02.885964 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:03.543198 (+657234us) log.cc:844] Serialized 14 byte log entry
0626 08:24:03.898894 (+355696us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:03.898945 (+ 51us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:03.899003 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:03.899102 (+ 99us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:03.899255 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32615,"prepare.queue_time_us":177267,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":142807,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":907826,"threads_started":2}
I20250626 08:24:03.938752 2582 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250626 08:24:03.939091 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.944200 1994 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250626 08:24:03.945485 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:03.807960 4558 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:03.878859 4062 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:03.844520 4599 raft_consensus.cc:738] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:03.896543 4439 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:03.978013 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.067824 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.064534 1926 raft_consensus.cc:2408] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20250626 08:24:03.928682 4557 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.273s user 0.008s sys 0.004s
I20250626 08:24:04.026026 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:03.925002 4189 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:04.003837 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:03.948894 4333 consensus_meta.cc:287] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.106s user 0.011s sys 0.000s
W20250626 08:24:03.939268 3886 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.485s user 0.006s sys 0.006s
I20250626 08:24:04.081260 1925 raft_consensus.cc:2408] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250626 08:24:04.083724 1847 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:04.081081 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.032529 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:03.942394 4560 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:04.022258 4681 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.021533 4545 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:04.113499 1846 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20250626 08:24:04.114534 1845 raft_consensus.cc:1238] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:04.115303 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.139398 1843 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
W20250626 08:24:04.132927 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.142165 4576 consensus_meta.cc:287] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.079s user 0.008s sys 0.003s
I20250626 08:24:04.142352 4519 consensus_queue.cc:1046] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:24:04.142603 4576 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:04.143500 4519 raft_consensus.cc:3053] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:04.143949 4519 raft_consensus.cc:738] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:04.144523 4576 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:04.144790 4519 consensus_queue.cc:260] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:04.077608 4664 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.146615 4599 consensus_queue.cc:260] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:04.147671 4599 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:04.150019 4334 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:04.142194 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 234063us
I20250626 08:24:03.911557 4595 consensus_queue.cc:1046] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.198s
I20250626 08:24:04.141777 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:04.155014 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 365012us
I20250626 08:24:04.156098 1928 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:04.156896 2582 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:04.076805 4609 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:04.163048 1893 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.159264 1926 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:04.160216 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.164892 2550 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:04.167138 2587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 653) took 1718 ms. Trace:
I20250626 08:24:04.118415 4545 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:04.054558 4361 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20250626 08:24:04.135157 4665 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:04.158407 1927 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.015s sys 0.023s
W20250626 08:24:04.170677 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.171335 1928 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20250626 08:24:04.169589 1816 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:04.185987 1927 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.015s sys 0.024s
I20250626 08:24:04.174345 1968 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.186857 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1958) took 1085 ms. Trace:
I20250626 08:24:04.187350 1927 rpcz_store.cc:276] 0626 08:24:03.101648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.101795 (+ 147us) service_pool.cc:225] Handling call
0626 08:24:03.125365 (+ 23570us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:03.125582 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:24:03.198232 (+ 72650us) log.cc:844] Serialized 14 byte log entry
0626 08:24:04.186387 (+988155us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:04.186462 (+ 75us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:04.186526 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.186634 (+ 108us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:04.186790 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":74886,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":87902,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":979146,"threads_started":2,"wal-append.queue_time_us":554270}
I20250626 08:24:04.169823 2587 rpcz_store.cc:276] 0626 08:24:02.448542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.448683 (+ 141us) service_pool.cc:225] Handling call
0626 08:24:02.565259 (+116576us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:02.589727 (+ 24468us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:02.589797 (+ 70us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:02.654048 (+ 64251us) log.cc:844] Serialized 52 byte log entry
0626 08:24:02.917356 (+263308us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:02.917409 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:02.917469 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:04.166942 (+1249473us) raft_consensus.cc:1683] finished
0626 08:24:04.166960 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:04.167085 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16136,"raft.queue_time_us":6135,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":34822,"thread_start_us":209498,"threads_started":1,"wal-append.queue_time_us":1020149}
I20250626 08:24:04.159046 4616 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:04.078126 4439 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:04.168808 4519 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:04.110141 4333 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.170130 4595 consensus_queue.cc:1230] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250626 08:24:04.227942 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.233057 1968 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.119607 3886 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.184891 4545 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.189767 4262 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250626 08:24:04.239673 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 48125.2us
W20250626 08:24:04.250164 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 47880.7us
I20250626 08:24:04.182579 4681 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:04.240850 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:04.253017 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 98605.2us
I20250626 08:24:04.259779 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.260601 1926 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:04.262531 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.256506 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.263901 1926 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.269752 1996 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:04.277046 1893 leader_election.cc:400] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
W20250626 08:24:04.277738 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 44744.5us
I20250626 08:24:04.277519 1893 leader_election.cc:403] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250626 08:24:04.236578 4573 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.234375 4062 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:04.282284 1841 raft_consensus.cc:2366] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5.
I20250626 08:24:04.284360 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.263675 4595 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:04.190258 4538 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20250626 08:24:04.254328 4262 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:04.263458 4681 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:04.292428 1967 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:04.315110 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.292544 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:04.293351 1968 connection.cc:232] RPC call timeout handler was delayed by -1.3262556029999359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1584s for a response
I20250626 08:24:04.213699 4616 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.317588 1994 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:04.313956 4262 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:04.329349 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.330265 1924 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:04.331084 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.332108 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.332530 4262 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.322160 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.336292 1994 raft_consensus.cc:2408] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250626 08:24:04.337527 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.338514 1994 raft_consensus.cc:2366] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 7.
I20250626 08:24:04.333134 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.340731 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.341722 1924 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:04.341781 4262 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:04.342517 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.344048 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.355217 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:04.359288 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44312.8us
I20250626 08:24:04.360378 1967 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.362301 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.363971 1994 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:04.365111 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.366109 1994 raft_consensus.cc:2391] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:04.368292 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.317092 4681 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.356848 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.314520 4361 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:04.371282 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 496) took 3380 ms. Trace:
I20250626 08:24:04.379348 1847 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:04.379707 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 210201us
W20250626 08:24:04.356730 4678 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.318s user 0.000s sys 0.001s
I20250626 08:24:04.317921 4538 consensus_queue.cc:1230] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250626 08:24:04.318918 4662 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 08:24:04.409901 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.426878 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.424337 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:04.410503 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.428665 2582 raft_consensus.cc:2391] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:04.345104 3988 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:24:04.411715 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 153907us
I20250626 08:24:04.418370 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.436347 1927 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:04.424865 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.427143 1846 raft_consensus.cc:2408] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250626 08:24:04.382206 1919 rpcz_store.cc:276] 0626 08:24:00.991109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.176974 (+185865us) service_pool.cc:225] Handling call
0626 08:24:01.177315 (+ 341us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:02.287697 (+1110382us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.371044 (+2083347us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400301954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:04.371227 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2552021248}
W20250626 08:24:04.432296 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.438340 1919 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:04.371253 4558 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.432014 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:04.433647 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:04.463227 1994 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:24:04.463083 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 50498.2us
I20250626 08:24:04.467830 1816 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.412199 4495 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.471129 1893 leader_election.cc:400] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:24:04.472939 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:04.463138 1968 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.584s (SENT) [suppressed 20 similar messages]
I20250626 08:24:04.466161 2587 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:04.470762 1816 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.476454 1919 raft_consensus.cc:1238] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:04.356572 4673 consensus_queue.cc:1046] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.286s
I20250626 08:24:04.481721 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:04.486665 1994 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:04.483057 1816 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.483546 1890 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:04.484164 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:04.435645 3988 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:04.501453 1847 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:04.429709 4662 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:04.491616 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 53408.3us
W20250626 08:24:04.497464 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.498174 1994 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:24:04.504592 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.439224 4687 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:24:04.490917 4679 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.410s user 0.000s sys 0.001s
I20250626 08:24:04.424731 4674 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.303s
I20250626 08:24:04.510533 1919 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250626 08:24:04.511548 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:04.515292 1994 raft_consensus.cc:1238] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:24:04.519047 1994 raft_consensus.cc:1238] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:04.520756 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 84782.6us
I20250626 08:24:04.498481 4189 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:04.517167 1919 pending_rounds.cc:85] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 3
I20250626 08:24:04.505734 3988 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:04.475426 4361 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20250626 08:24:04.524197 1919 pending_rounds.cc:107] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250626 08:24:04.524621 1928 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:24:04.509394 4486 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.607s user 0.013s sys 0.000s
I20250626 08:24:04.526032 1919 raft_consensus.cc:2887] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:04.526091 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.513563 4662 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:04.529804 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:04.531489 1928 raft_consensus.cc:1238] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:04.508744 4683 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.536125 1928 raft_consensus.cc:1273] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:24:04.523106 3886 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.535246 4687 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:04.527135 4439 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:04.542752 4715 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250626 08:24:04.552173 4715 raft_consensus.cc:3053] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:04.552568 4715 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:04.552899 4706 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250626 08:24:04.555723 1919 raft_consensus.cc:1273] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250626 08:24:04.543190 4682 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.329s user 0.001s sys 0.000s
I20250626 08:24:04.543407 1893 leader_election.cc:400] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:04.532127 4697 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:04.543493 4699 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:04.544065 4696 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.525384 4189 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:04.525592 4333 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.500958 4659 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.261s
W20250626 08:24:04.543699 4713 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250626 08:24:04.558828 1893 leader_election.cc:403] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20250626 08:24:04.575564 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36150 (request call id 655) took 1481 ms (client timeout 1870 ms). Trace:
I20250626 08:24:04.575690 4616 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:04.576053 2579 rpcz_store.cc:269] 0626 08:24:03.094586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.100819 (+ 6233us) service_pool.cc:225] Handling call
0626 08:24:04.575310 (+1474491us) spinlock_profiling.cc:243] Waited 79.4 ms on lock 0x7b54005eae54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:04.575483 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182560640}
I20250626 08:24:04.577376 1893 leader_election.cc:400] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:04.577661 1893 leader_election.cc:403] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:04.578094 4558 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:04.564332 4700 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:04.580641 2583 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
W20250626 08:24:04.564553 4500 consensus_meta.cc:287] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.732s user 0.010s sys 0.004s
W20250626 08:24:04.566329 4693 log.cc:927] Time spent T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
W20250626 08:24:04.567065 4709 log.cc:927] Time spent T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
W20250626 08:24:04.568375 4496 consensus_meta.cc:287] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.737s user 0.007s sys 0.004s
W20250626 08:24:04.569164 4703 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20250626 08:24:04.569554 4189 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:04.570284 4692 log.cc:927] Time spent T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.252s user 0.000s sys 0.001s
I20250626 08:24:04.561812 4439 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:04.530503 4486 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:04.571396 4695 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.226s user 0.001s sys 0.000s
W20250626 08:24:04.565250 4702 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
W20250626 08:24:04.587194 4649 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.977s user 0.009s sys 0.002s
I20250626 08:24:04.581620 4675 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:24:04.581950 4558 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.592610 2579 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:45504
W20250626 08:24:04.599172 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.603559 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.559319 4694 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 2. Current term is 3.
I20250626 08:24:04.613983 1926 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:04.615661 3988 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:04.627246 1994 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:04.627305 4545 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.617134 4712 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.091s
I20250626 08:24:04.580554 4697 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:24:04.589370 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1969) took 1087 ms. Trace:
I20250626 08:24:04.620918 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.621348 1919 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:04.629058 2587 raft_consensus.cc:2391] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:04.630131 1926 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:04.633730 1918 rpcz_store.cc:276] 0626 08:24:03.502445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.948698 (+446253us) service_pool.cc:225] Handling call
0626 08:24:04.053159 (+104461us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:04.053597 (+ 438us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:04.053642 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:04.081278 (+ 27636us) log.cc:844] Serialized 26 byte log entry
0626 08:24:04.242365 (+161087us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:24:04.242429 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.242525 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:04.589160 (+346635us) raft_consensus.cc:1683] finished
0626 08:24:04.589184 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:04.589311 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13757,"raft.queue_time_us":100575,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":23753,"thread_start_us":91611,"threads_started":1,"wal-append.queue_time_us":170335}
I20250626 08:24:04.579236 4700 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.553417 4715 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.594348 4609 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
W20250626 08:24:04.673669 4667 consensus_meta.cc:287] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.764s user 0.007s sys 0.002s
I20250626 08:24:04.669481 1842 raft_consensus.cc:3053] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:04.698210 1842 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:04.672472 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 59348.2us
W20250626 08:24:04.676201 4698 log.cc:927] Time spent T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.242s user 0.003s sys 0.000s
I20250626 08:24:04.691025 4700 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.628059 3988 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:04.722014 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 84455.2us
I20250626 08:24:04.717651 4496 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:04.722846 4496 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.723887 4707 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:04.724813 4496 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:04.614675 4705 consensus_queue.cc:1046] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20250626 08:24:04.726629 4603 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:04.727445 4716 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:04.728097 4603 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:04.728215 4716 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:04.728602 4603 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:04.695362 4687 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:04.699057 1842 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.731148 4697 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:04.731528 1842 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.717221 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:04.735371 1893 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:04.736510 1893 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:04.719761 4545 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:04.720707 4262 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:24:04.721539 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2117) took 1003 ms. Trace:
I20250626 08:24:04.656179 2579 raft_consensus.cc:604] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.622795 4649 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:04.594816 4699 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:04.717924 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 54419.5us
I20250626 08:24:04.740938 4690 raft_consensus.cc:2762] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20250626 08:24:04.740940 4687 raft_consensus.cc:2762] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250626 08:24:04.743198 4694 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:24:04.744240 2586 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:04.769099 2000 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:04.729527 4716 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.748574 1927 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 9.
W20250626 08:24:04.750015 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 389731us
I20250626 08:24:04.750417 1919 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:04.751066 1893 leader_election.cc:400] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:04.786360 1893 leader_election.cc:403] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:04.750816 2001 rpcz_store.cc:276] 0626 08:24:03.732304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.743567 (+ 11263us) service_pool.cc:225] Handling call
0626 08:24:03.743904 (+ 337us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:03.750020 (+ 6116us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:03.750065 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:03.766002 (+ 15937us) log.cc:844] Serialized 26 byte log entry
0626 08:24:04.171959 (+405957us) raft_consensus.cc:1643] Marking committed up to 1
0626 08:24:04.172000 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.172073 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:04.664122 (+492049us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b4800042fd8. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebdf36871 00007f4ebdf2da17 00007f4ebdf2d8aa 00007f4ebe02297f 00007f4ebe022647 00007f4ebe036c6c 00007f4ebe036c18 00007f4ebe036b98 00007f4ebe036b53 00007f4ebe035dfc 00007f4ebcfbc956
0626 08:24:04.673496 (+ 9374us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:04.721318 (+ 47822us) raft_consensus.cc:1683] finished
0626 08:24:04.721341 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:04.721479 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59046,"raft.queue_time_us":49266,"raft.run_cpu_time_us":2780,"raft.run_wall_time_us":870742,"spinlock_wait_cycles":49651840,"thread_start_us":377702,"threads_started":1,"wal-append.queue_time_us":491566}
I20250626 08:24:04.722209 4486 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:04.758112 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.751066 4634 consensus_meta.cc:287] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.983s user 0.011s sys 0.000s
W20250626 08:24:04.765208 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.785671 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:04.791520 2550 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:04.792236 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:04.793195 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.729491 4603 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:04.797706 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 334576us
I20250626 08:24:04.799294 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:04.799870 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.801432 2001 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:04.770354 4662 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.809638 2001 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250626 08:24:04.802321 2586 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250626 08:24:04.803762 1842 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250626 08:24:04.811724 2586 pending_rounds.cc:85] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting all ops after (but not including) 5
I20250626 08:24:04.812153 1842 pending_rounds.cc:85] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting all ops after (but not including) 6
I20250626 08:24:04.812227 2586 pending_rounds.cc:107] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250626 08:24:04.812435 1842 pending_rounds.cc:107] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250626 08:24:04.812553 2586 raft_consensus.cc:2887] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:04.812718 1842 raft_consensus.cc:2887] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:04.819794 1968 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.821375 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.821441 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.825187 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.825894 1927 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:04.827495 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.829591 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 696) took 1077 ms. Trace:
I20250626 08:24:04.798547 4697 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:04.829998 2586 rpcz_store.cc:276] 0626 08:24:03.752267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.752481 (+ 214us) service_pool.cc:225] Handling call
0626 08:24:03.766279 (+ 13798us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.744033 (+977754us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b5800854fb0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb8ff9e73 00007f4eb905d1a7 00007f4eb9075bd7 00007f4eb90773ce 00007f4eb9081729 00007f4eb9081ff6 00007f4eb907e97f 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744
0626 08:24:04.829336 (+ 85303us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.829529 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2184826112}
W20250626 08:24:04.834679 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 87018.2us
I20250626 08:24:04.835404 1893 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:04.835404 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.802445 4486 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:04.883015 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:04.865468 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:04.830629 4625 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250626 08:24:04.885627 1844 raft_consensus.cc:3053] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:04.832927 2001 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20250626 08:24:04.892854 2001 pending_rounds.cc:85] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 7
I20250626 08:24:04.887064 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1768) took 2275 ms. Trace:
I20250626 08:24:04.893191 2001 pending_rounds.cc:107] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20250626 08:24:04.797821 4609 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20250626 08:24:04.891621 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 508) took 2050 ms. Trace:
I20250626 08:24:04.802286 4612 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:04.892787 1844 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:04.893554 2001 raft_consensus.cc:2887] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:04.840984 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:04.893240 1848 rpcz_store.cc:276] 0626 08:24:02.611539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.692813 (+ 81274us) service_pool.cc:225] Handling call
0626 08:24:03.116515 (+423702us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:03.117202 (+ 687us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.886879 (+1769677us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540032f5d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:04.887009 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929911680}
W20250626 08:24:04.910250 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 69379.9us
I20250626 08:24:04.834358 4603 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.902290 1921 rpcz_store.cc:276] 0626 08:24:02.841636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.841753 (+ 117us) service_pool.cc:225] Handling call
0626 08:24:02.883113 (+ 41360us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:02.883305 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 8
0626 08:24:02.918186 (+ 34881us) log.cc:844] Serialized 14 byte log entry
0626 08:24:02.918607 (+ 421us) log.cc:844] Serialized 14 byte log entry
0626 08:24:04.666422 (+1747815us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b18003b79a0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:04.891032 (+224610us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b18003b79a0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611d555 00007f4eb9076c50 00007f4eb90892f6 00007f4eb9099e67
0626 08:24:04.891162 (+ 130us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:04.891230 (+ 68us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:04.891294 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.891394 (+ 100us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:04.891539 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25224,"prepare.queue_time_us":256321,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":14610,"raft.queue_time_us":24509,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":19231,"spinlock_wait_cycles":53115904,"thread_start_us":98076,"threads_started":1}
I20250626 08:24:04.916190 2586 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
mode: GRACEFUL
new_leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
from {username='slave'} at 127.0.0.1:45504
I20250626 08:24:04.802736 4545 consensus_queue.cc:1046] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20250626 08:24:04.796072 4634 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:04.919631 1927 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:05.007349 2586 raft_consensus.cc:604] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Received request to transfer leadership to TS 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:24:05.010592 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.012488 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:04.882923 4694 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:05.025496 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.837671 4697 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:05.017747 2002 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.017s sys 0.015s
I20250626 08:24:04.883175 4570 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.018968 2586 raft_consensus.cc:612] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:05.019312 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:04.930148 2001 raft_consensus.cc:1273] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:05.020432 1967 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:04.840126 4496 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:05.038703 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.017s sys 0.047s
W20250626 08:24:05.039767 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.017s sys 0.047s
I20250626 08:24:04.830194 4723 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:04.911120 4656 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250626 08:24:05.031363 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 196018us
W20250626 08:24:05.042070 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:05.032939 2002 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.018s sys 0.015s
I20250626 08:24:04.885025 4558 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:04.889884 4649 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:04.889097 4333 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:04.897352 4561 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:05.056506 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:04.832777 4667 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.190s
I20250626 08:24:05.028208 4612 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:05.038297 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:04.888653 4699 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.043879 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.038559 4697 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:04.890466 4663 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
I20250626 08:24:04.917865 4722 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:05.046900 1843 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:05.038945 4700 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:24:05.052443 1965 connection.cc:232] RPC call timeout handler was delayed by -1.0587257150000369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.263s for a response
W20250626 08:24:05.023252 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.008s sys 0.026s
I20250626 08:24:04.932560 1844 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.034868 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2127) took 1181 ms. Trace:
I20250626 08:24:05.060683 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.043936 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2123) took 1217 ms. Trace:
I20250626 08:24:04.912089 4189 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.073305 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.100435 1847 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:05.074811 1928 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:05.045324 4684 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:05.078686 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.084087 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.101260 1842 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:05.094377 1844 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:05.101414 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:05.102296 1927 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:24:05.095613 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.009s sys 0.026s
W20250626 08:24:05.103624 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 620607us
I20250626 08:24:05.096680 1893 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.091159 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.094663 1965 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:05.098452 2002 rpcz_store.cc:276] 0626 08:24:03.826548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.826681 (+ 133us) service_pool.cc:225] Handling call
0626 08:24:03.827016 (+ 335us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:04.171811 (+344795us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:04.171990 (+ 179us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:04.240235 (+ 68245us) log.cc:844] Serialized 52 byte log entry
0626 08:24:04.338609 (+ 98374us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:04.377665 (+ 39056us) log.cc:844] Serialized 14 byte log entry
0626 08:24:05.043290 (+665625us) log.cc:844] Serialized 14 byte log entry
0626 08:24:05.043527 (+ 237us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.043629 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:05.043686 (+ 57us) raft_consensus.cc:1683] finished
0626 08:24:05.043704 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.043859 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66724,"prepare.queue_time_us":131998,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":182499,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":9985,"spinlock_wait_cycles":21159168,"thread_start_us":1029112,"threads_started":4,"wal-append.queue_time_us":72116}
I20250626 08:24:05.105468 1843 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:24:05.083529 4334 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.021s sys 0.075s
I20250626 08:24:05.082147 4558 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.117632 2002 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:05.119356 2002 raft_consensus.cc:1238] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
W20250626 08:24:05.123718 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.025s sys 0.042s
W20250626 08:24:05.123791 2585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 593) took 3354 ms (client timeout 2630 ms). Trace:
W20250626 08:24:05.124087 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.859s user 0.025s sys 0.042s
W20250626 08:24:05.124277 2585 rpcz_store.cc:269] 0626 08:24:01.769759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:01.769898 (+ 139us) service_pool.cc:225] Handling call
0626 08:24:05.123590 (+3353692us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b54005e8154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.123717 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857337600}
I20250626 08:24:05.127490 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:05.124437 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 859202us
I20250626 08:24:05.136368 1927 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
W20250626 08:24:05.114264 4334 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s user 0.021s sys 0.076s
I20250626 08:24:05.140084 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.144667 1847 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:05.138064 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:05.147748 1842 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 9.
I20250626 08:24:05.151863 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:05.130676 4633 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 2.229s user 0.022s sys 0.056s
I20250626 08:24:05.134382 4558 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.123301 4700 raft_consensus.cc:695] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:05.154266 4633 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 2.261s user 0.022s sys 0.057s
I20250626 08:24:05.130870 4640 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:05.166139 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:05.163195 1967 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:05.154702 4599 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.126058 4673 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20250626 08:24:05.132222 4694 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:05.136273 4712 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.032s
I20250626 08:24:05.168301 4667 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election won for term 3
W20250626 08:24:05.160758 4681 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.748s user 0.006s sys 0.006s
I20250626 08:24:05.166824 4662 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.182543 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:05.220170 1968 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:05.136524 4724 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:05.161497 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 64906.2us
I20250626 08:24:05.186151 4649 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:05.187310 4558 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:05.187745 4699 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:05.188620 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 61274.1us
I20250626 08:24:05.097896 1993 rpcz_store.cc:276] 0626 08:24:03.852172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.883930 (+ 31758us) service_pool.cc:225] Handling call
0626 08:24:03.884390 (+ 460us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:03.884853 (+ 463us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:03.884907 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:03.885492 (+ 585us) log.cc:844] Serialized 26 byte log entry
0626 08:24:04.248668 (+363176us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:04.476239 (+227571us) log.cc:844] Serialized 14 byte log entry
0626 08:24:04.882403 (+406164us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.885604 (+ 3201us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:04.914877 (+ 29273us) raft_consensus.cc:1683] finished
0626 08:24:04.914907 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.033661 (+118754us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":209694,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":40880,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":3661,"thread_start_us":771576,"threads_started":3,"wal-append.queue_time_us":494917}
I20250626 08:24:05.191494 4729 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:05.193795 1814 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:05.240703 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:05.158931 4711 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
W20250626 08:24:05.255244 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 277638us
W20250626 08:24:05.260738 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 67044.3us
I20250626 08:24:05.242264 4640 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.230283 4634 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.260656 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.184500 4599 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.262231 1814 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:05.251313 4496 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:05.253180 4663 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.230113 4681 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.276721 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.289783 1842 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.265022 4558 raft_consensus.cc:2762] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250626 08:24:05.268059 4519 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:05.267892 4189 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.275408 4727 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.305740 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250626 08:24:05.307246 4736 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250626 08:24:05.288661 4731 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
W20250626 08:24:05.314119 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 91636.1us
I20250626 08:24:05.315027 1893 leader_election.cc:400] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5.
I20250626 08:24:05.315709 4727 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:05.315955 1893 leader_election.cc:400] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:05.316838 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:05.317713 4694 raft_consensus.cc:2762] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250626 08:24:05.331403 4599 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.274354 4496 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:05.333964 4496 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:05.337538 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.339895 4697 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.294636 4663 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.344583 4663 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.350483 4673 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:24:05.259897 4700 consensus_queue.cc:237] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.352721 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.303435 4519 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.329705 1844 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:05.354050 1999 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:05.354882 4681 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.355868 4712 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.304719 4612 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:05.358144 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 490) took 2458 ms. Trace:
I20250626 08:24:05.358950 1893 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:05.356725 4519 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:05.365991 1844 rpcz_store.cc:276] 0626 08:24:02.899903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.973251 (+ 73348us) service_pool.cc:225] Handling call
0626 08:24:03.013449 (+ 40198us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.885528 (+1872079us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54000a2e20. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900e2f0 00007f4eb9080da0 00007f4eb9081e11 00007f4eb907e97f 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c
0626 08:24:05.357473 (+471945us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.358008 (+ 535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117989632}
I20250626 08:24:05.377465 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.373493 1842 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:05.378577 1844 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:05.381597 1893 leader_election.cc:400] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 7.
I20250626 08:24:05.382900 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.379673 1968 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:05.376154 4593 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250626 08:24:05.376375 4671 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:05.376703 4558 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.407737 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.412758 1893 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.415091 1967 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:05.416765 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.394294 4640 raft_consensus.cc:2802] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:05.420176 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.421694 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:05.424415 1926 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:05.424616 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.406378 4671 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:05.410472 4681 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.380396 4712 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.450526 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.452853 1967 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:05.415249 4694 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:05.418264 4724 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:05.418208 4699 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.415153 4662 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.403350 4667 raft_consensus.cc:695] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:05.422299 4697 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:05.460008 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 172517us
I20250626 08:24:05.460541 4189 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:05.440057 4558 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.452723 4633 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.439110 4640 raft_consensus.cc:695] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:05.472195 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:05.473536 4746 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:05.474805 4667 consensus_queue.cc:237] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.479291 4189 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:05.480051 4694 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:05.480506 4694 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.480876 4694 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:05.482035 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.477411 4711 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250626 08:24:05.483646 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.484477 2001 raft_consensus.cc:2391] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:05.484824 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2136) took 1535 ms. Trace:
I20250626 08:24:05.485225 1994 rpcz_store.cc:276] 0626 08:24:03.949074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.524068 (+574994us) service_pool.cc:225] Handling call
0626 08:24:04.524332 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.627794 (+103462us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.484523 (+856729us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54001310d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.484742 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236202880}
I20250626 08:24:05.487152 1814 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:05.487257 4643 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:05.488126 4727 consensus_queue.cc:1035] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:24:05.488581 4749 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:05.489162 4749 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:05.470309 4735 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:05.490610 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1972) took 1318 ms. Trace:
I20250626 08:24:05.490999 1924 rpcz_store.cc:276] 0626 08:24:04.172689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.352944 (+180255us) service_pool.cc:225] Handling call
0626 08:24:04.363470 (+ 10526us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.697275 (+333805us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:04.697326 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:04.697387 (+ 61us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:04.697442 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.490346 (+792904us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54001aa754. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.490399 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.490539 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":767022848}
I20250626 08:24:05.491901 1927 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:24:05.492002 1926 raft_consensus.cc:1273] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250626 08:24:05.493009 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 377) took 2740 ms. Trace:
I20250626 08:24:05.469404 4662 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.493384 1995 rpcz_store.cc:276] 0626 08:24:02.752316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.754421 (+ 2105us) service_pool.cc:225] Handling call
0626 08:24:02.776248 (+ 21827us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:02.945164 (+168916us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:02.945191 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:02.959294 (+ 14103us) log.cc:844] Serialized 26 byte log entry
0626 08:24:03.475664 (+516370us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:03.475722 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.492652 (+2016930us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b54000d07d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.492696 (+ 44us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:05.492741 (+ 45us) raft_consensus.cc:1683] finished
0626 08:24:05.492753 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.492953 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40507264,"thread_start_us":373865,"threads_started":1,"wal-append.queue_time_us":619971,"wal-append.run_cpu_time_us":2913,"wal-append.run_wall_time_us":1277382}
I20250626 08:24:05.470050 4699 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.495534 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2115) took 1795 ms. Trace:
W20250626 08:24:05.496372 1920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1837) took 3485 ms (client timeout 2455 ms). Trace:
I20250626 08:24:05.478896 4633 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:05.496788 1920 rpcz_store.cc:269] 0626 08:24:02.010899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.026058 (+ 15159us) service_pool.cc:225] Handling call
0626 08:24:05.496152 (+3470094us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54001a9d54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.496291 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246004096}
I20250626 08:24:05.497115 1921 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:05.498226 4724 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.498420 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1957) took 2397 ms. Trace:
I20250626 08:24:05.460143 4681 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.498804 4724 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.497666 4640 consensus_queue.cc:237] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.500905 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.501538 1918 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:05.502678 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:05.503274 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.502727 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1841) took 1711 ms. Trace:
I20250626 08:24:05.504135 1925 rpcz_store.cc:276] 0626 08:24:03.791336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.931647 (+140311us) service_pool.cc:225] Handling call
0626 08:24:05.502458 (+1570811us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b54002b0054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.502645 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6134016}
W20250626 08:24:05.503752 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.503168 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 491) took 2603 ms. Trace:
I20250626 08:24:05.507733 1849 rpcz_store.cc:276] 0626 08:24:02.900197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.980095 (+ 79898us) service_pool.cc:225] Handling call
0626 08:24:02.980477 (+ 382us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:03.220119 (+239642us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.502966 (+2282847us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54000352d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.503116 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492051840}
I20250626 08:24:05.508975 4712 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.509078 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 380) took 2735 ms. Trace:
I20250626 08:24:05.509433 1996 rpcz_store.cc:276] 0626 08:24:02.774111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.774536 (+ 425us) service_pool.cc:225] Handling call
0626 08:24:02.789233 (+ 14697us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.279203 (+1489970us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.508866 (+1229663us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400203fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.509021 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3398470016}
I20250626 08:24:05.510926 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 384) took 1440 ms. Trace:
I20250626 08:24:05.511231 2002 raft_consensus.cc:2366] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:05.497439 4671 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:05.512316 4671 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:05.514007 4645 raft_consensus.cc:1062] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6: attempting to promote NON_VOTER be56082cacb54e9aa8f1a9ee91e70d76 to VOTER
I20250626 08:24:05.514972 4749 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.518131 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.518743 2001 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:05.518770 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.519517 1846 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:05.519799 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.520949 1842 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:05.522135 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.511276 1999 rpcz_store.cc:276] 0626 08:24:04.070808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.669041 (+598233us) service_pool.cc:225] Handling call
0626 08:24:04.669420 (+ 379us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:05.357557 (+688137us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.510725 (+153168us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b540039ce54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.510872 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1573902720}
I20250626 08:24:05.532182 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.532325 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:05.532723 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.532831 1925 raft_consensus.cc:1273] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:05.533807 1967 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:05.534111 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.535224 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.535310 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.536481 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.537593 4652 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.538401 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.540571 4585 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
W20250626 08:24:05.541790 1841 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1772) took 2929 ms (client timeout 1584 ms). Trace:
I20250626 08:24:05.532827 2001 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:05.542582 1843 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:05.543287 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.544127 1845 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:05.545825 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1789) took 1109 ms. Trace:
I20250626 08:24:05.546156 1845 rpcz_store.cc:276] 0626 08:24:04.436346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.436572 (+ 226us) service_pool.cc:225] Handling call
0626 08:24:05.543967 (+1107395us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:05.544470 (+ 503us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.545654 (+ 1184us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540038c050. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.545773 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2546043776}
W20250626 08:24:05.542191 1841 rpcz_store.cc:269] 0626 08:24:02.612638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.640924 (+ 28286us) service_pool.cc:225] Handling call
0626 08:24:05.541561 (+2900637us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540038c054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.541700 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3714724096}
I20250626 08:24:05.495891 2000 rpcz_store.cc:276] 0626 08:24:03.700392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.736646 (+ 36254us) service_pool.cc:225] Handling call
0626 08:24:03.751281 (+ 14635us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:04.769525 (+1018244us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:05.495330 (+725805us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400131ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.495485 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2340255488}
I20250626 08:24:05.551079 1842 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:05.498816 1923 rpcz_store.cc:276] 0626 08:24:03.101106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.112149 (+ 11043us) service_pool.cc:225] Handling call
0626 08:24:03.112587 (+ 438us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:03.580371 (+467784us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:03.580404 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:03.603773 (+ 23369us) log.cc:844] Serialized 52 byte log entry
0626 08:24:04.120666 (+516893us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:04.166551 (+ 45885us) log.cc:844] Serialized 14 byte log entry
0626 08:24:04.192844 (+ 26293us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:04.689000 (+496156us) spinlock_profiling.cc:243] Waited 9.58 ms on lock 0x7b18002d5fe0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb612060f 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:05.498124 (+809124us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54002e04d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:05.498188 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:05.498232 (+ 44us) raft_consensus.cc:1683] finished
0626 08:24:05.498246 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.498368 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51779,"prepare.queue_time_us":46980,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":7500,"raft.queue_time_us":164805,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":31070,"spinlock_wait_cycles":1037375616,"thread_start_us":469726,"threads_started":1,"wal-append.queue_time_us":611613}
I20250626 08:24:05.570045 4745 consensus_queue.cc:1046] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20250626 08:24:05.567796 1814 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:05.571060 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.573328 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.571789 1919 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:05.574450 1996 raft_consensus.cc:2408] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:24:05.566890 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.575937 1994 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:05.571759 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.578361 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.579138 1999 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:05.580139 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.580905 1923 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:05.583146 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.583791 1919 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:05.584933 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.585481 1919 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:05.585479 1967 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:05.577562 1849 raft_consensus.cc:2374] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7. Re-sending same reply.
I20250626 08:24:05.587066 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:05.587487 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.587934 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:05.588392 1995 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:05.588541 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.589187 1846 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:05.586380 1814 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:05.595973 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.487267 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:05.548579 4662 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.596824 1841 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:05.570588 4699 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.560495 4724 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.613194 4614 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:05.582732 4599 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:05.599309 4570 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:05.600111 4733 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:05.622437 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.595783 4746 raft_consensus.cc:3053] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:24:05.595538 4745 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:05.568245 4663 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20250626 08:24:05.620829 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.571009 4056 consensus_queue.cc:1046] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20250626 08:24:05.603754 4652 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.612568 4633 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.603928 4665 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:05.706383 4733 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.585623 4731 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20250626 08:24:05.714247 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.639690 4599 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.640074 4570 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.584164 4539 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
W20250626 08:24:05.726768 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 195353us
I20250626 08:24:05.706923 4746 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:05.708880 4189 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.728277 1923 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:05.710018 4745 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.724643 4665 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.718649 4681 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.732692 4599 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.729835 4684 raft_consensus.cc:3053] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:05.731197 4733 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.735553 4570 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:05.724152 4652 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.750749 4745 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:05.734580 4643 raft_consensus.cc:3053] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:05.744572 1967 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
W20250626 08:24:05.735071 1893 connection.cc:232] RPC call timeout handler was delayed by -2.5796591610001087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20250626 08:24:05.785626 1893 connection.cc:232] RPC call timeout handler was delayed by -2.1809217800000624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20250626 08:24:05.781344 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2121) took 2013 ms. Trace:
I20250626 08:24:05.786906 1997 rpcz_store.cc:276] 0626 08:24:03.768313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.790350 (+ 22037us) service_pool.cc:225] Handling call
0626 08:24:03.826010 (+ 35660us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:03.826366 (+ 356us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:03.826407 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:03.826464 (+ 57us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:03.826508 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:03.826584 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:05.781080 (+1954496us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b540039c450. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:05.781262 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6,"raft.queue_time_us":53536,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":134,"spinlock_wait_cycles":32614144}
I20250626 08:24:05.755985 4665 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.764657 4684 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:05.752143 4189 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.795413 4558 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:05.797055 4640 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:05.858414 4640 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:05.858850 4640 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:05.859961 4578 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:05.864384 1924 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250626 08:24:05.865381 4657 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.865927 4657 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.866262 4663 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:24:05.868722 4496 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.534s user 0.006s sys 0.003s
I20250626 08:24:05.789705 4643 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:05.879987 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.869161 4496 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:05.882149 4754 log.cc:927] Time spent T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250626 08:24:05.882833 4657 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:05.883615 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:05.884619 4643 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.885150 4496 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:05.885886 4643 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:05.886435 4652 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:05.886592 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:05.871039 4558 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.892130 4570 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.894436 4684 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.800014 4519 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.900437 4712 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:05.906785 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 991620us
I20250626 08:24:05.908188 1850 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:05.908787 4545 raft_consensus.cc:3053] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:05.908715 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:05.911355 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.923404 1967 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:05.911054 4665 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:05.912945 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:05.926899 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:05.946285 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.950307 1841 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250626 08:24:05.958055 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:05.961037 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.962939 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:05.914947 4486 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:05.934752 4545 raft_consensus.cc:738] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:05.976008 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:05.976791 1893 connection.cc:232] RPC call timeout handler was delayed by -1.149006145000385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
I20250626 08:24:05.933542 4613 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.933140 4397 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:05.980795 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:05.968765 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:05.952379 4758 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
I20250626 08:24:05.972573 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:05.987380 1918 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:24:05.990449 2585 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:05.970997 4686 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.001s sys 0.012s
I20250626 08:24:05.960522 4707 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.989245 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:05.967432 4684 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:05.992753 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.021106 1918 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:05.993330 2550 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:05.892163 4694 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.019977 1926 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:06.033809 4545 consensus_queue.cc:260] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.960196 4671 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
W20250626 08:24:06.035058 4732 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.776s user 0.004s sys 0.000s
I20250626 08:24:06.035784 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:06.058420 4486 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.799010 4746 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:06.017755 4603 raft_consensus.cc:668] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:24:05.948159 4712 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:05.975373 4614 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:05.969391 4778 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:06.048591 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 67895.5us
I20250626 08:24:06.091383 1918 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:06.018563 4308 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:06.055979 4545 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:06.057746 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:05.988864 4613 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:05.984933 4719 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:05.993386 4686 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.212s user 0.002s sys 0.012s
I20250626 08:24:06.040207 4733 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.065295 1926 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250626 08:24:06.110827 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.069607 2585 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:24:06.070540 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 77296.2us
I20250626 08:24:05.987250 4716 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:06.071203 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.029453 4727 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:06.086064 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 123224us
W20250626 08:24:06.087611 1998 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2110) took 2389 ms (client timeout 1915 ms). Trace:
W20250626 08:24:06.117731 1998 rpcz_store.cc:269] 0626 08:24:03.697748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:03.702288 (+ 4540us) service_pool.cc:225] Handling call
0626 08:24:06.087356 (+2385068us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b540039d354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:06.087487 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21726976}
I20250626 08:24:06.043555 4735 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.122277 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.091284 4585 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:06.090654 1920 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250626 08:24:06.099016 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.017374 4612 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.097s
W20250626 08:24:06.103067 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.063499 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.044591 4707 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.086251 4582 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:06.082798 4772 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:06.086299 4671 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:06.120456 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.128145 1893 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.137630 1849 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.079241 4640 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.111161 4308 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:06.141039 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.138722 4705 consensus_queue.cc:1046] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250626 08:24:06.142226 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:06.098327 4731 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250626 08:24:06.149760 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.168073 1925 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.136219 4582 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:06.154399 4764 log.cc:927] Time spent T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250626 08:24:06.154933 4189 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.155135 1893 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.155265 4712 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.098117 4778 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.117192 4397 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250626 08:24:06.157398 4733 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.209852 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.210703 1921 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.096388 4614 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.160259 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.157765 4716 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250626 08:24:06.182157 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 68541.1us
I20250626 08:24:06.174979 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.181404 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:06.220155 2550 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:06.220531 2585 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
W20250626 08:24:06.220973 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 102146us
I20250626 08:24:06.136758 4771 raft_consensus.cc:2762] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20250626 08:24:06.158223 4772 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:06.215750 1918 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:06.215761 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:06.218231 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 63164.2us
I20250626 08:24:06.114852 4719 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.135658 4585 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.225126 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.225693 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.127195 4727 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.224843 4308 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.167186 4570 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.230192 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.205847 4633 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:06.231830 1926 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:06.204021 4665 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.166798 4558 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.143114 4671 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.239789 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.182294 4582 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:06.241144 1924 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.231310 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.241324 1849 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:06.211616 4724 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.245501 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.249658 1923 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:06.250193 4724 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.251276 2586 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:06.252811 4724 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.183743 4519 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.215667 4699 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.255153 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.256291 2585 raft_consensus.cc:2408] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250626 08:24:06.174527 4599 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:06.259634 1814 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:06.265709 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:06.268155 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.135737 4735 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.268949 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.235836 4705 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:06.275823 1847 raft_consensus.cc:2366] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:06.272909 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.279616 1842 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:06.235664 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:06.242905 4772 raft_consensus.cc:695] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:06.278190 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:06.275983 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 54478.4us
I20250626 08:24:06.243479 4776 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:06.282800 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.283497 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.278061 1968 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:06.222223 4769 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:06.286566 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:06.285802 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 64653.6us
I20250626 08:24:06.274056 4189 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:06.290858 1968 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:06.287165 1844 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:06.282356 4694 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.279147 4765 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:06.282711 4705 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.300060 1925 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:06.307664 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.302029 1924 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:06.259764 4699 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.310108 1814 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:06.289232 4189 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.305595 1921 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:06.311844 1965 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.276461 4771 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.310108 1890 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:06.258699 4633 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.294202 4665 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:06.291491 2579 raft_consensus.cc:3058] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.313584 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:06.297818 4765 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:06.311187 4767 log.cc:927] Time spent T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20250626 08:24:06.330446 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.288327 4776 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:06.327492 1814 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:06.332199 2581 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:06.311218 4705 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:06.334357 2580 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:06.336935 4765 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.314162 4189 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:06.341172 4539 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:06.334012 4792 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:06.346493 4772 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.313474 4774 raft_consensus.cc:2762] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250626 08:24:06.314728 4749 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:06.352802 1965 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.367797 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 703) took 1937 ms. Trace:
I20250626 08:24:06.383534 2582 rpcz_store.cc:276] 0626 08:24:04.430739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.439443 (+ 8704us) service_pool.cc:225] Handling call
0626 08:24:04.439833 (+ 390us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:04.516402 (+ 76569us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:04.516436 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:05.815436 (+1299000us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b1800436d80. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb7a226d1 00007f4eb7a31d02 00007f4eb90bac51 00007f4eb907f02a
0626 08:24:05.822280 (+ 6844us) log.cc:844] Serialized 26 byte log entry
0626 08:24:06.085756 (+263476us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:06.085786 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.085839 (+ 53us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:06.362576 (+276737us) raft_consensus.cc:1683] finished
0626 08:24:06.362596 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:06.367719 (+ 5123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71550,"raft.queue_time_us":51866,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":4686,"spinlock_wait_cycles":6092416,"thread_start_us":219507,"threads_started":1,"wal-append.queue_time_us":349315}
I20250626 08:24:06.337128 4776 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.385931 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.386554 1924 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:06.368319 4707 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.386503 1965 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:06.369597 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 488) took 3470 ms (client timeout 2185 ms). Trace:
W20250626 08:24:06.394094 1850 rpcz_store.cc:269] 0626 08:24:02.899310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:02.962718 (+ 63408us) service_pool.cc:225] Handling call
0626 08:24:06.369429 (+3406711us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54000343d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:06.369527 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6680816896}
I20250626 08:24:06.388198 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:06.397490 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:06.355298 4785 log.cc:927] Time spent T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20250626 08:24:06.341989 4775 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
W20250626 08:24:06.405613 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 174575us
I20250626 08:24:06.294716 4769 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.356151 4640 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.411602 2581 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:06.416931 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.427853 2002 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:24:06.429461 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.431143 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.431881 1925 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:24:06.395830 4539 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:06.391572 4665 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:06.373381 4792 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.436914 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.437036 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:06.407397 4614 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.407625 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.439608 2002 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:06.419998 4724 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:06.464187 1918 raft_consensus.cc:2466] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:06.472504 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.426579 4775 raft_consensus.cc:695] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:06.467700 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.505298 1925 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:06.440423 4795 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.072s user 0.002s sys 0.000s
I20250626 08:24:06.501564 4539 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.436645 4578 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:06.431394 4749 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:06.439986 4699 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.410481 4519 consensus_queue.cc:1046] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20250626 08:24:06.514459 2581 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:06.406440 4712 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:06.514266 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 77351.9us
I20250626 08:24:06.536160 1925 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:06.458968 4671 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.469024 4599 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.555318 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.438926 4792 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:06.559243 1967 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:06.559408 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:06.468415 4797 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:06.581040 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:06.580112 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 292088us
I20250626 08:24:06.582074 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:06.578876 4735 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:06.554134 4643 consensus_meta.cc:287] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.668s user 0.009s sys 0.002s
I20250626 08:24:06.588034 1846 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250626 08:24:06.588457 1846 pending_rounds.cc:85] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting all ops after (but not including) 2
I20250626 08:24:06.588708 1846 pending_rounds.cc:107] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250626 08:24:06.588987 1846 raft_consensus.cc:2887] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:06.540843 4578 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:06.564779 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.543030 4778 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.533972 4798 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250626 08:24:06.558494 4712 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.608840 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.610785 4794 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20250626 08:24:06.516006 4719 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.613210 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.559979 4799 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:06.619343 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:06.601044 4578 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:06.633329 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 336855us
I20250626 08:24:06.600029 4633 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.633268 4799 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.563851 4664 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
W20250626 08:24:06.630199 4745 consensus_meta.cc:287] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.830s user 0.009s sys 0.002s
I20250626 08:24:06.620225 4671 raft_consensus.cc:2802] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
W20250626 08:24:06.649816 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.651161 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:06.635659 4798 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:06.668506 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.670054 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:06.670938 1918 raft_consensus.cc:2391] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:06.619467 4707 consensus_queue.cc:1046] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20250626 08:24:06.672811 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:06.591107 4665 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.676983 4806 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.645139 4799 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:06.677047 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.691812 4694 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:06.692783 4781 consensus_queue.cc:1046] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:06.693199 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:06.693895 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.694244 4694 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:06.694792 4694 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:06.694819 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.695550 1997 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:06.674541 4671 raft_consensus.cc:695] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:06.644421 4308 consensus_queue.cc:1035] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20250626 08:24:06.709131 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.709338 1849 raft_consensus.cc:1238] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:06.709852 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.710975 2551 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:06.675701 4633 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:06.715642 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.695324 4694 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:06.680675 4778 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:06.722905 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:06.724903 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:06.726541 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.725546 1843 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 3: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:06.703363 4707 raft_consensus.cc:3053] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:24:06.637940 4749 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.730713 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.651135 4745 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.733072 1844 raft_consensus.cc:2374] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7. Re-sending same reply.
I20250626 08:24:06.736186 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:06.737793 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:06.740998 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:06.695326 4781 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20250626 08:24:06.725384 4769 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.738615 4664 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:24:06.742003 4782 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.293s user 0.000s sys 0.001s
I20250626 08:24:06.731912 4707 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:06.744665 4520 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.754858 4811 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:06.755002 4769 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250626 08:24:06.755275 4790 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.306s user 0.001s sys 0.000s
W20250626 08:24:06.755582 4802 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
W20250626 08:24:06.758229 4800 log.cc:927] Time spent T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.108s user 0.001s sys 0.001s
I20250626 08:24:06.758982 4803 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:06.645963 4775 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.751437 4804 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.751338 4781 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:06.770953 4811 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:06.754344 4664 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.787047 4745 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:06.788148 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:06.788269 4804 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:06.792804 4684 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.749s user 0.010s sys 0.002s
I20250626 08:24:06.706867 4806 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.769661 4671 consensus_queue.cc:237] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:06.801107 4545 consensus_meta.cc:287] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.696s user 0.010s sys 0.001s
I20250626 08:24:06.801973 1843 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250626 08:24:06.802340 1844 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20250626 08:24:06.803599 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 62701.3us
I20250626 08:24:06.829592 1893 leader_election.cc:400] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:06.829885 1893 leader_election.cc:403] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:06.802350 4811 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.830821 1893 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:06.831162 4705 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.831542 4811 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.833213 4705 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.804121 4189 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.836048 4189 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.836755 4189 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:06.837472 4189 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:06.839392 2579 raft_consensus.cc:2466] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:06.766372 1993 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:06.821786 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:06.831674 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:06.814744 4664 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.856261 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:06.859447 1994 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:06.826058 4712 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:06.867611 1923 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.067s sys 0.070s
W20250626 08:24:06.868031 1923 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.067s sys 0.070s
I20250626 08:24:06.857015 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:06.863830 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:06.868705 1995 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:06.864441 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.869292 1893 leader_election.cc:400] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:06.869591 1893 leader_election.cc:403] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20250626 08:24:06.865790 4771 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.536s user 0.011s sys 0.000s
W20250626 08:24:06.866003 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.62288e+06us
I20250626 08:24:06.872143 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:06.867251 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:06.875243 4595 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:06.875742 4595 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.876087 4595 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:06.876309 4803 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:06.858315 2580 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:06.875097 4778 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:06.874248 1996 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:06.869653 1999 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:06.837860 4189 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:06.885471 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2002) took 1006 ms. Trace:
I20250626 08:24:06.887132 1927 rpcz_store.cc:276] 0626 08:24:05.879441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.921847 (+ 42406us) service_pool.cc:225] Handling call
0626 08:24:05.925475 (+ 3628us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:05.969218 (+ 43743us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:05.969252 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:05.969754 (+ 502us) log.cc:844] Serialized 26 byte log entry
0626 08:24:06.092301 (+122547us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:06.092340 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.092400 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:06.885224 (+792824us) raft_consensus.cc:1683] finished
0626 08:24:06.885253 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:06.885401 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":978,"raft.queue_time_us":44856,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"thread_start_us":165368,"threads_started":2,"wal-append.queue_time_us":255956}
I20250626 08:24:06.860680 4769 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:06.907014 4673 consensus_queue.cc:1035] T 28139bf29a374c98a8fec4eddf814fac P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250626 08:24:06.920819 1995 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:06.928997 1994 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
W20250626 08:24:06.934952 4582 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.666s user 0.008s sys 0.002s
I20250626 08:24:06.935510 4582 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.843201 4707 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.937700 4582 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.881681 4771 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:06.886581 4189 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:06.916891 4769 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:06.961871 4776 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.551s user 0.002s sys 0.009s
I20250626 08:24:06.962872 4776 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:06.964051 1925 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:06.964102 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1811) took 1093 ms. Trace:
I20250626 08:24:06.966562 4804 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.967976 4776 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.961664 2578 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:06.974565 4694 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:06.975771 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:06.978432 2587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 704) took 2453 ms (client timeout 2289 ms). Trace:
W20250626 08:24:06.979084 2587 rpcz_store.cc:269] 0626 08:24:04.525385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.525549 (+ 164us) service_pool.cc:225] Handling call
0626 08:24:06.978160 (+2452611us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b54005438d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:06.978345 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":224937216}
I20250626 08:24:06.964502 1846 rpcz_store.cc:276] 0626 08:24:05.870376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.870518 (+ 142us) service_pool.cc:225] Handling call
0626 08:24:05.893524 (+ 23006us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:06.589437 (+695913us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.963853 (+374416us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b5400210f54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:06.964043 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1596214016}
W20250626 08:24:06.987356 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 484820us
I20250626 08:24:06.978556 4694 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:06.999090 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 520) took 2436 ms. Trace:
I20250626 08:24:06.999675 1928 rpcz_store.cc:276] 0626 08:24:04.562903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.582151 (+ 19248us) service_pool.cc:225] Handling call
0626 08:24:04.582536 (+ 385us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:05.107470 (+524934us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.998724 (+1891254us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54001a8e54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:06.998998 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1131575168}
W20250626 08:24:07.004172 4779 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.837s user 0.004s sys 0.000s
I20250626 08:24:07.012799 2584 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:07.014411 4613 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:06.876859 4803 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.017297 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:07.022338 2585 raft_consensus.cc:2374] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7. Re-sending same reply.
I20250626 08:24:07.018168 4803 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:07.020759 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:07.037500 2578 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:07.039014 4731 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:06.938855 4699 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:07.036422 4806 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:07.041396 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:07.042276 1890 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:07.043306 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 93952.9us
I20250626 08:24:06.945353 4778 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:07.050179 1995 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:06.918638 4519 consensus_queue.cc:1046] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250626 08:24:07.046896 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:24:07.040048 4723 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:07.053980 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:07.059479 1926 raft_consensus.cc:2391] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:07.050477 1814 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:07.066752 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:07.029417 4707 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:06.948474 4629 consensus_queue.cc:1046] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20250626 08:24:07.061556 4803 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:07.028517 4769 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:07.159767 1994 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:07.044776 4712 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:07.170360 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 127497us
I20250626 08:24:07.106809 4519 consensus_queue.cc:1230] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer 755d83ca0913406b816e95415d502055 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250626 08:24:07.109668 4806 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:07.192998 4799 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.170517 4803 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:07.057119 4699 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:07.177707 4673 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:07.198511 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:07.107306 4792 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.057780 4731 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.180279 4712 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.220973 1926 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 8. Current term is 9. Ops: []
W20250626 08:24:07.420933 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.012s sys 0.012s
W20250626 08:24:07.519616 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.802s user 0.014s sys 0.014s
I20250626 08:24:07.535086 1928 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:07.537678 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:07.538235 4686 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:07.289435 4727 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:07.270100 1997 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:07.539000 4686 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.273105 4649 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:07.540356 4649 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:24:07.541759 4686 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:07.268380 4633 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:07.544637 4633 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.548182 4633 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:07.305778 4578 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.660s user 0.009s sys 0.000s
I20250626 08:24:07.576676 4486 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:07.576596 4578 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.289932 3886 raft_consensus.cc:3053] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
W20250626 08:24:07.253191 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 55510.1us
I20250626 08:24:07.578614 3886 raft_consensus.cc:738] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:07.290676 4595 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.574829 4707 consensus_queue.cc:1046] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20250626 08:24:07.422224 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.050s sys 0.073s
W20250626 08:24:07.605675 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.053s sys 0.076s
W20250626 08:24:07.306847 4765 consensus_meta.cc:287] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.936s user 0.011s sys 0.000s
W20250626 08:24:07.393438 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.011s sys 0.079s
W20250626 08:24:07.627336 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.907s user 0.011s sys 0.086s
W20250626 08:24:07.308176 4520 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.532s user 0.009s sys 0.001s
I20250626 08:24:07.597162 2001 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250626 08:24:07.607661 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:07.621387 2581 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.035s sys 0.093s
W20250626 08:24:07.628412 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 917390us
I20250626 08:24:07.628826 1926 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:07.628844 2581 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.937s user 0.035s sys 0.093s
I20250626 08:24:07.251693 1993 raft_consensus.cc:2408] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250626 08:24:07.574359 4649 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:07.223623 4803 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:07.183149 4723 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:07.423241 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.046s sys 0.063s
W20250626 08:24:07.787389 1841 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.037s sys 0.064s
W20250626 08:24:07.788267 1841 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s user 0.038s sys 0.064s
I20250626 08:24:07.752545 4812 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:07.754667 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.360s user 0.007s sys 0.035s
W20250626 08:24:07.803205 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.409s user 0.007s sys 0.035s
W20250626 08:24:07.803851 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.4173e+06us
I20250626 08:24:07.608336 4727 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:07.804550 1965 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:07.665124 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:07.805712 2001 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:24:07.757463 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 128276us
W20250626 08:24:07.444397 4700 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.025s sys 0.025s
W20250626 08:24:07.758810 1812 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.000s sys 0.023s
W20250626 08:24:07.807202 1812 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.992s user 0.000s sys 0.024s
W20250626 08:24:07.663336 4539 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.008s sys 0.020s
I20250626 08:24:07.753149 4599 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:07.807477 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 992945us
I20250626 08:24:07.622424 4715 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:07.808656 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:07.821735 1968 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:07.786736 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.095s user 0.052s sys 0.070s
I20250626 08:24:07.826313 2001 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
W20250626 08:24:07.766283 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 133661us
W20250626 08:24:07.633471 4189 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.675s user 0.012s sys 0.000s
I20250626 08:24:07.197484 4806 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.605085 4673 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:07.809434 1998 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.248s user 0.025s sys 0.024s
W20250626 08:24:07.865381 1998 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.304s user 0.025s sys 0.024s
I20250626 08:24:07.759191 4520 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:07.807014 4700 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.398s user 0.029s sys 0.034s
I20250626 08:24:07.831743 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:07.872717 2581 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 2.
W20250626 08:24:07.812740 4539 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s user 0.008s sys 0.025s
W20250626 08:24:07.863487 4632 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.012s sys 0.028s
W20250626 08:24:07.859865 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 55391us
I20250626 08:24:07.663240 4633 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:07.862877 1926 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:24:07.824585 4772 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.982s user 0.004s sys 0.031s
I20250626 08:24:07.854880 4799 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:07.870258 2551 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:07.833770 4599 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.818217 4729 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:07.834446 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:07.877101 1927 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:07.906837 1965 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:07.934319 2001 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:07.304423 4664 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.866516 3886 consensus_queue.cc:260] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.909787 4585 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:07.916949 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:07.948314 1926 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:07.884975 4813 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:07.935845 1965 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:07.633994 4765 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.812999 4812 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:07.832304 4649 consensus_queue.cc:1046] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20250626 08:24:07.814545 4727 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:07.931977 4772 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.105s user 0.004s sys 0.033s
W20250626 08:24:07.906404 4632 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s user 0.012s sys 0.028s
I20250626 08:24:07.872058 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:07.909068 4728 consensus_queue.cc:1046] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:07.955575 3886 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:07.956646 1996 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.010s sys 0.019s
I20250626 08:24:07.859277 4189 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.869589 4486 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20250626 08:24:07.888968 4809 consensus_queue.cc:1046] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20250626 08:24:07.967303 1918 raft_consensus.cc:1238] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250626 08:24:07.936820 4539 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:07.940624 4578 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:07.967028 1965 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:08.009815 1965 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:08.012108 1965 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:07.984469 1996 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.878s user 0.011s sys 0.019s
I20250626 08:24:07.949838 4712 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:07.967273 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.068s user 0.013s sys 0.017s
W20250626 08:24:08.018796 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.119s user 0.013s sys 0.017s
I20250626 08:24:07.956909 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:07.932797 4633 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:08.019533 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.15026e+06us
I20250626 08:24:07.956099 4723 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:07.967373 4749 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.004s sys 0.025s
I20250626 08:24:07.933969 4613 consensus_queue.cc:1046] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20250626 08:24:08.038973 1893 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:08.022199 4486 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:07.975162 4727 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:07.966285 4823 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:08.083461 2001 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:07.976356 2578 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:08.015695 4189 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:08.122201 4189 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:08.122828 4189 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:07.970119 4729 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:08.124675 4729 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:07.966532 4799 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:08.086730 4749 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s user 0.011s sys 0.029s
I20250626 08:24:07.975692 4772 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:08.088766 4827 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20250626 08:24:08.086447 4723 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:08.090165 4808 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250626 08:24:08.090761 4595 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:08.091670 4829 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20250626 08:24:08.006789 1926 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:07.973176 4649 raft_consensus.cc:3053] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:08.019479 4813 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:08.141368 4649 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:08.033200 4792 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:08.093736 4731 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:08.113986 4609 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.066s user 0.058s sys 0.078s
I20250626 08:24:08.120574 4712 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:08.087879 4699 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:08.109638 4769 consensus_meta.cc:287] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.930s user 0.011s sys 0.000s
I20250626 08:24:08.026866 4539 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.141556 4813 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:08.115832 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:08.016734 4673 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:08.123515 4632 raft_consensus.cc:3053] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:07.105058 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:24:08.128914 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2151) took 1061 ms. Trace:
I20250626 08:24:08.370786 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2158) took 1194 ms. Trace:
W20250626 08:24:08.368986 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 330053us
I20250626 08:24:08.371544 2001 rpcz_store.cc:276] 0626 08:24:07.176680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.115545 (+938865us) service_pool.cc:225] Handling call
0626 08:24:08.370704 (+255159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:08.318542 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2015) took 2058 ms. Trace:
I20250626 08:24:08.373819 1923 rpcz_store.cc:276] 0626 08:24:06.260370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.260962 (+ 592us) service_pool.cc:225] Handling call
0626 08:24:06.261386 (+ 424us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:06.261803 (+ 417us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:06.261846 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:06.279613 (+ 17767us) log.cc:844] Serialized 26 byte log entry
0626 08:24:06.870517 (+590904us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:06.870576 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.870666 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:08.318298 (+1447632us) raft_consensus.cc:1683] finished
0626 08:24:08.318320 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:08.318479 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10787,"thread_start_us":559737,"threads_started":1,"wal-append.queue_time_us":1223046}
W20250626 08:24:08.313176 1741 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32850 (request call id 306) took 23422 ms (client timeout 15000 ms). Trace:
I20250626 08:24:08.378995 1893 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:08.233636 4595 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:08.144763 4649 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.394587 4595 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.395059 4649 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:08.395522 4539 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:08.396113 4813 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:08.396512 4595 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:08.129557 4823 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:08.139907 4728 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Stepping down as leader of term 8
I20250626 08:24:08.398229 4728 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:08.398826 4823 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:08.345461 4712 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:08.399288 4728 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:08.370586 4632 raft_consensus.cc:738] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:08.139962 4772 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:08.345196 4609 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s user 0.064s sys 0.079s
W20250626 08:24:08.314025 4816 log.cc:927] Time spent T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
W20250626 08:24:08.266552 4826 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20250626 08:24:08.318958 4765 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:08.318542 4599 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:08.228842 4806 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:08.314674 4633 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:08.376066 1741 rpcz_store.cc:269] 0626 08:23:44.890315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:23:44.894057 (+ 3742us) service_pool.cc:225] Handling call
0626 08:24:08.313080 (+23419023us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:24:01.243227 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:24:01.244013 (+ 786us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:24:01.244078 (+ 65us) write_op.cc:435] Acquired schema lock
0626 08:24:01.244105 (+ 27us) tablet.cc:660] Decoding operations
0626 08:24:01.245150 (+ 1045us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:24:01.245224 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0626 08:24:01.245256 (+ 32us) tablet.cc:683] Acquiring locks for 6 operations
0626 08:24:01.269405 (+ 24149us) tablet.cc:699] Row locks acquired
0626 08:24:01.271207 (+ 1802us) write_op.cc:260] PREPARE: finished
0626 08:24:01.275519 (+ 4312us) write_op.cc:270] Start()
0626 08:24:01.275666 (+ 147us) write_op.cc:276] Timestamp: P: 1750926241275440 usec, L: 0
0626 08:24:01.275686 (+ 20us) op_driver.cc:348] REPLICATION: starting
0626 08:24:01.276424 (+ 738us) log.cc:844] Serialized 2462 byte log entry
0626 08:24:02.594865 (+1318441us) op_driver.cc:464] REPLICATION: finished
0626 08:24:03.042400 (+447535us) write_op.cc:301] APPLY: starting
0626 08:24:03.086105 (+ 43705us) tablet.cc:1365] starting BulkCheckPresence
0626 08:24:03.086326 (+ 221us) tablet.cc:1368] finished BulkCheckPresence
0626 08:24:03.086481 (+ 155us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:24:03.087608 (+ 1127us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:24:03.087893 (+ 285us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0626 08:24:03.087957 (+ 64us) write_op.cc:312] APPLY: finished
0626 08:24:03.088274 (+ 317us) log.cc:844] Serialized 53 byte log entry
0626 08:24:03.088443 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:03.088557 (+ 114us) write_op.cc:454] Released schema lock
0626 08:24:03.088622 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0626 08:24:03.204719 (+116097us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:03.220194 (+ 15475us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":104578,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":379885,"apply.run_cpu_time_us":5853,"apply.run_wall_time_us":117174,"num_ops":6,"prepare.queue_time_us":112828,"prepare.run_cpu_time_us":4400,"prepare.run_wall_time_us":33557,"replication_time_us":1319015,"thread_start_us":267668,"threads_started":2}]]}
I20250626 08:24:08.319133 4664 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:08.269345 4812 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:08.416605 4715 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.584s user 0.010s sys 0.001s
I20250626 08:24:08.431690 4774 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:08.401237 4769 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:08.229022 4707 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:08.421026 4189 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:08.412322 1993 rpcz_store.cc:276] 0626 08:24:07.067516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.862496 (+794980us) service_pool.cc:225] Handling call
0626 08:24:08.126296 (+263800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:08.400408 4712 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:08.436578 4609 raft_consensus.cc:3053] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
W20250626 08:24:08.440541 2579 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.041s sys 0.024s
I20250626 08:24:08.453593 1925 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
W20250626 08:24:08.442273 4775 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.253s user 0.035s sys 0.060s
I20250626 08:24:08.441097 4581 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:08.445369 4665 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.160s user 0.027s sys 0.050s
I20250626 08:24:08.453747 4609 raft_consensus.cc:738] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:08.452708 4774 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:08.497121 4774 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.497290 4609 consensus_queue.cc:260] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.497051 4189 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:08.498384 4609 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:08.498785 4189 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:08.498098 2580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 621) took 2186 ms (client timeout 2142 ms). Trace:
W20250626 08:24:08.499621 2580 rpcz_store.cc:269] 0626 08:24:06.312020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.382528 (+ 70508us) service_pool.cc:225] Handling call
0626 08:24:08.497869 (+2115341us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b540050da54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:08.498022 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":872539904}
I20250626 08:24:08.499766 4581 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:08.499187 4189 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:08.502955 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2148) took 1438 ms. Trace:
I20250626 08:24:08.500223 4769 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:08.503976 2586 raft_consensus.cc:2408] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250626 08:24:08.506146 1993 rpcz_store.cc:276] 0626 08:24:07.064770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.453106 (+1388336us) service_pool.cc:225] Handling call
0626 08:24:08.502414 (+ 49308us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.502615 (+ 201us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:08.502646 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:08.502701 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:08.502742 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:08.502803 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:08.502903 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:08.492018 2579 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.273s user 0.041s sys 0.025s
W20250626 08:24:08.518435 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.036s sys 0.030s
W20250626 08:24:08.494791 4775 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.305s user 0.035s sys 0.060s
W20250626 08:24:08.504030 4665 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.219s user 0.027s sys 0.050s
W20250626 08:24:08.520736 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.024s sys 0.033s
W20250626 08:24:08.523584 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.024s sys 0.033s
W20250626 08:24:08.518954 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.761s user 0.036s sys 0.030s
I20250626 08:24:08.499272 4774 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:08.525367 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 767976us
W20250626 08:24:08.525755 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:08.525650 1993 raft_consensus.cc:1238] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
W20250626 08:24:08.522037 1891 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.031s sys 0.026s
W20250626 08:24:08.540633 1891 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.885s user 0.031s sys 0.026s
I20250626 08:24:08.540684 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2149) took 1475 ms. Trace:
W20250626 08:24:08.523032 1892 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.019s sys 0.043s
I20250626 08:24:08.445870 4707 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:08.541710 1892 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.911s user 0.019s sys 0.043s
I20250626 08:24:08.541121 1921 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:08.539886 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:08.541169 1993 rpcz_store.cc:276] 0626 08:24:07.065480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.508147 (+1442667us) service_pool.cc:225] Handling call
0626 08:24:08.525395 (+ 17248us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.540418 (+ 15023us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:08.540622 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:08.518918 4712 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:08.442202 4632 consensus_queue.cc:260] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:08.547508 4772 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:08.574975 1841 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.008s sys 0.033s
I20250626 08:24:08.550346 4599 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:08.546219 4308 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:24:08.575501 1841 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.757s user 0.008s sys 0.033s
I20250626 08:24:08.563266 4774 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:08.558149 4629 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:08.570736 4807 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.006s sys 0.033s
I20250626 08:24:08.561304 4749 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:08.605453 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 508) took 1897 ms. Trace:
I20250626 08:24:08.561023 4775 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:08.565372 4832 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:08.564615 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.004s sys 0.008s
W20250626 08:24:08.603075 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.029s sys 0.037s
W20250626 08:24:08.708496 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.004s sys 0.021s
W20250626 08:24:08.563082 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.028s sys 0.014s
W20250626 08:24:09.215054 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.428s user 0.028s sys 0.039s
I20250626 08:24:08.577950 4772 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:08.682420 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:08.653694 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:09.209568 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.192s user 0.014s sys 0.026s
W20250626 08:24:08.977007 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.013s sys 0.012s
W20250626 08:24:09.002153 1926 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.002s sys 0.014s
W20250626 08:24:08.993959 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.167s user 0.003s sys 0.027s
I20250626 08:24:08.577685 4632 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:08.575850 4715 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:09.214465 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.260s user 0.009s sys 0.026s
W20250626 08:24:09.245715 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.37643e+06us
I20250626 08:24:09.247103 2551 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:09.249110 2551 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:09.216200 2001 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.000s sys 0.011s
W20250626 08:24:09.250666 2001 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.000s sys 0.011s
W20250626 08:24:08.927050 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:08.809799 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:08.733166 1998 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.031s sys 0.023s
W20250626 08:24:09.253278 1998 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.343s user 0.039s sys 0.034s
I20250626 08:24:08.578071 4599 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:09.215546 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.393s user 0.044s sys 0.051s
I20250626 08:24:09.257582 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:08.609447 4807 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.805s user 0.006s sys 0.033s
W20250626 08:24:09.258462 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:09.260555 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:08.606199 4731 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:09.061282 4803 consensus_meta.cc:287] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.251s user 0.013s sys 0.000s
I20250626 08:24:09.222339 4775 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.261368 4731 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.261440 4803 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.222177 4832 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.263785 4803 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.264318 4832 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.129420 4842 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:09.075835 4649 consensus_meta.cc:287] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.680s user 0.009s sys 0.002s
I20250626 08:24:09.264400 4803 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:08.693084 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.002s sys 0.023s
W20250626 08:24:08.979714 4833 log.cc:927] Time spent T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.402s user 0.001s sys 0.000s
I20250626 08:24:08.609190 4774 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:09.075073 4707 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.057346 4839 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:09.259905 4836 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.199s user 0.002s sys 0.000s
W20250626 08:24:09.361104 1926 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.218s user 0.009s sys 0.018s
W20250626 08:24:09.360196 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.448s user 0.005s sys 0.030s
I20250626 08:24:08.608407 1841 rpcz_store.cc:276] 0626 08:24:06.708114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.708252 (+ 138us) service_pool.cc:225] Handling call
0626 08:24:06.725305 (+ 17053us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:06.725644 (+ 339us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:06.725674 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:06.726129 (+ 455us) log.cc:844] Serialized 26 byte log entry
0626 08:24:07.788806 (+1062677us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:08.592187 (+803381us) log.cc:844] Serialized 14 byte log entry
0626 08:24:08.605119 (+ 12932us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:08.605205 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:08.605248 (+ 43us) raft_consensus.cc:1683] finished
0626 08:24:08.605261 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:08.605396 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12607,"raft.queue_time_us":20266,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":22791,"thread_start_us":1827907,"threads_started":2,"wal-append.queue_time_us":1303207}
W20250626 08:24:09.301991 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 52959.8us
W20250626 08:24:09.221652 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:09.364810 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.538s user 0.005s sys 0.034s
I20250626 08:24:09.265820 1928 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:09.223958 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 1992) took 3684 ms. Trace:
W20250626 08:24:09.360491 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.508s user 0.013s sys 0.025s
I20250626 08:24:09.243634 4539 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.399513 1920 rpcz_store.cc:276] 0626 08:24:05.539294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.569231 (+ 29937us) service_pool.cc:225] Handling call
0626 08:24:05.587723 (+ 18492us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:06.132871 (+545148us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.223621 (+3090750us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b540030eb54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:09.223874 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1155849984}
W20250626 08:24:09.401407 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.57442e+06us
W20250626 08:24:09.259732 4831 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.545s user 0.001s sys 0.002s
W20250626 08:24:09.258797 4189 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.758s user 0.011s sys 0.002s
I20250626 08:24:09.215303 4581 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:08.716866 4700 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.009s sys 0.017s
W20250626 08:24:09.403802 4625 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.002s sys 0.014s
W20250626 08:24:09.403965 4673 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.021s sys 0.030s
W20250626 08:24:09.322721 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.008s sys 0.034s
W20250626 08:24:08.773700 4834 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
I20250626 08:24:08.604638 4308 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:09.362008 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.34987e+06us
I20250626 08:24:08.610098 4629 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:09.265327 4835 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.276s user 0.001s sys 0.002s
I20250626 08:24:09.100571 4838 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.316359 4840 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.260444 4599 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:09.428756 4625 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.711s user 0.002s sys 0.015s
I20250626 08:24:09.265246 4841 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.450508 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:09.451354 2583 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:09.377897 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.360970 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:09.260636 4715 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.423748 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:09.455842 1920 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:09.383819 4707 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:08.629799 4749 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:09.428419 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.048s user 0.013s sys 0.034s
W20250626 08:24:09.463207 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.08436e+06us
I20250626 08:24:09.464170 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:09.362421 4826 log.cc:927] Time spent T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250626 08:24:09.434376 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:09.465623 2581 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:09.377120 4803 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.292420 4842 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:09.428859 4673 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.029s user 0.022s sys 0.030s
I20250626 08:24:08.712467 4486 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:09.427874 4700 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.492s user 0.014s sys 0.022s
I20250626 08:24:09.362071 4772 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:09.453518 1993 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.010s sys 0.018s
W20250626 08:24:09.440742 4837 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
I20250626 08:24:09.473865 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:09.452759 1999 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:09.417157 4539 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.460773 2587 raft_consensus.cc:1273] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:09.386852 4839 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:09.471487 1993 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.011s sys 0.018s
W20250626 08:24:09.472227 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 338898us
I20250626 08:24:09.439896 4581 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
W20250626 08:24:09.475319 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2143) took 2415 ms (client timeout 1835 ms). Trace:
W20250626 08:24:09.495203 1999 rpcz_store.cc:269] 0626 08:24:07.060367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.060511 (+ 144us) service_pool.cc:225] Handling call
0626 08:24:09.377708 (+2317197us) spinlock_profiling.cc:243] Waited 83.8 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:09.475239 (+ 97531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192723328}
I20250626 08:24:09.474772 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:09.496688 2583 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:09.495108 1893 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:09.426120 4189 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.475044 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2144) took 2413 ms. Trace:
I20250626 08:24:09.500223 1996 rpcz_store.cc:276] 0626 08:24:07.061646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.061814 (+ 168us) service_pool.cc:225] Handling call
0626 08:24:07.062467 (+ 653us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:07.062711 (+ 244us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:07.062750 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:07.072972 (+ 10222us) log.cc:844] Serialized 26 byte log entry
0626 08:24:08.015205 (+942233us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:08.071998 (+ 56793us) log.cc:844] Serialized 14 byte log entry
0626 08:24:08.105749 (+ 33751us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:08.105858 (+ 109us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:09.474841 (+1368983us) raft_consensus.cc:1683] finished
0626 08:24:09.474865 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:09.474989 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":61806,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":38870,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":908483,"threads_started":1,"wal-append.queue_time_us":1417823}
I20250626 08:24:09.502856 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.503566 1996 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:09.459440 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:09.453819 4262 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.022s sys 0.067s
I20250626 08:24:09.450356 4852 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.451881 4840 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.481860 2550 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:09.475868 4486 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.531042 4486 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.483347 4673 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.531764 4673 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.525779 4852 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.485164 4842 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.486866 4848 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.487221 4772 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.488023 4645 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.479779 4700 consensus_queue.cc:1035] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20250626 08:24:09.471613 2578 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:09.445015 4838 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.514221 2582 raft_consensus.cc:3053] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 08:24:09.548072 2582 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:09.515792 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.516568 4803 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:09.518040 1997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 397) took 2811 ms (client timeout 1767 ms). Trace:
I20250626 08:24:09.479344 4853 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:09.527904 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 46094.3us
W20250626 08:24:09.570735 1997 rpcz_store.cc:269] 0626 08:24:06.707112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.707252 (+ 140us) service_pool.cc:225] Handling call
0626 08:24:09.517764 (+2810512us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54002805d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:09.517969 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1247656832}
I20250626 08:24:09.480806 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:09.572547 1920 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:09.537449 2583 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
W20250626 08:24:09.518639 4262 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.916s user 0.023s sys 0.067s
I20250626 08:24:09.549079 1999 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:09.550240 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:09.578172 1924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1878) took 3167 ms (client timeout 1530 ms). Trace:
W20250626 08:24:09.578787 1924 rpcz_store.cc:269] 0626 08:24:06.410592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.411233 (+ 641us) service_pool.cc:225] Handling call
0626 08:24:09.550067 (+3138834us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:09.578075 (+ 28008us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38664960}
I20250626 08:24:09.551359 4774 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.508421 4558 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:09.522217 4775 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.522843 4664 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:09.554466 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.558667 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 608) took 3573 ms. Trace:
I20250626 08:24:09.534567 4848 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.562752 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 666) took 2832 ms. Trace:
I20250626 08:24:09.534291 4842 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:09.566370 4733 raft_consensus.cc:3053] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:09.567487 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 712) took 2987 ms. Trace:
I20250626 08:24:09.656086 2586 rpcz_store.cc:276] 0626 08:24:06.580581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.580758 (+ 177us) service_pool.cc:225] Handling call
0626 08:24:09.567278 (+2986520us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400045954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:09.567424 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4420004608}
I20250626 08:24:09.521116 4581 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:09.469338 4841 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.533875 4673 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.578001 2582 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.587882 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:09.589501 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.596757 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:09.668457 1920 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:09.534919 4852 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:09.569418 4486 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:09.585986 4558 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:09.646709 2584 rpcz_store.cc:276] 0626 08:24:05.985761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.985961 (+ 200us) service_pool.cc:225] Handling call
0626 08:24:07.012502 (+1026541us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:07.013391 (+ 889us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.558463 (+2545072us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54005438d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:09.558606 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2277544192}
W20250626 08:24:09.660276 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.579634 4803 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:09.667956 1997 raft_consensus.cc:3058] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:09.669627 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:09.685035 2550 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 2.035s (SENT) [suppressed 9 similar messages]
I20250626 08:24:09.592381 4774 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:09.647099 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2156) took 2567 ms (client timeout 1533 ms). Trace:
I20250626 08:24:09.652506 4772 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:09.655715 4624 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:09.694216 1994 rpcz_store.cc:269] 0626 08:24:07.079738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.178492 (+ 98754us) service_pool.cc:225] Handling call
0626 08:24:09.554249 (+2375757us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:09.647011 (+ 92762us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139770112}
I20250626 08:24:09.694836 2582 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:09.704634 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.582118 4853 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.658599 4664 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:09.659204 4733 raft_consensus.cc:738] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:09.707013 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.710635 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 542) took 2175 ms. Trace:
W20250626 08:24:09.737610 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.671200 4842 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:24:09.693043 4787 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 2.224s user 0.012s sys 0.015s
W20250626 08:24:09.693409 4846 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20250626 08:24:09.709589 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.700153 4486 raft_consensus.cc:695] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:09.704474 4838 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.723410 2582 raft_consensus.cc:1273] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250626 08:24:09.725658 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.728868 1996 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:09.794508 1994 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:09.694561 4558 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:09.714105 1999 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:09.706858 4863 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:09.736819 1926 rpcz_store.cc:276] 0626 08:24:07.534749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.141178 (+606429us) service_pool.cc:225] Handling call
0626 08:24:08.142620 (+ 1442us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.142822 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:09.384957 (+1242135us) log.cc:844] Serialized 14 byte log entry
0626 08:24:09.710199 (+325242us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:09.710271 (+ 72us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:09.710330 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.710418 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:09.710558 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1307557,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":1566593,"threads_started":2}
I20250626 08:24:09.799803 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.692947 4856 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:09.745674 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 346824us
I20250626 08:24:09.800872 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.801558 1999 raft_consensus.cc:2391] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:09.697992 4640 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:09.755440 3988 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:09.808446 4840 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.808270 4581 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.755679 4496 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:09.810176 4841 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.810508 4496 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.758776 1965 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:09.812242 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 53571.2us
W20250626 08:24:09.774612 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.721529 1812 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:09.790015 1814 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:09.710397 4629 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.702437 4774 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.731438 1920 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:09.824949 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2045) took 1133 ms. Trace:
W20250626 08:24:09.738610 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:09.717442 4624 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.741758 1997 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:24:09.803599 2551 connection.cc:232] RPC call timeout handler was delayed by -1.5177750689999812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20250626 08:24:09.802673 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.806625 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:09.814312 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.794411 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 622) took 3482 ms. Trace:
I20250626 08:24:09.748888 4700 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:09.708580 4772 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.700912 4803 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.790056 4681 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.798494 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.825438 1920 rpcz_store.cc:276] 0626 08:24:08.691620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.573365 (+881745us) service_pool.cc:225] Handling call
0626 08:24:09.824861 (+251496us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:09.805675 4603 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250626 08:24:09.820871 4189 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.380264 4875 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.380890 4189 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:10.380968 4875 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.381500 4189 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:10.214541 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.383742 4189 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.384264 4189 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.384703 4189 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.385319 4875 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.344257 4848 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.345794 4853 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.387164 4774 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:10.387851 4876 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250626 08:24:10.388736 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2001) took 4570 ms. Trace:
I20250626 08:24:10.387917 4774 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.391243 1926 raft_consensus.cc:2408] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20250626 08:24:10.391872 1926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2037) took 1831 ms (client timeout 2216 ms). Trace:
W20250626 08:24:10.392408 1926 rpcz_store.cc:269] 0626 08:24:08.560489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.800081 (+1239592us) service_pool.cc:225] Handling call
0626 08:24:10.391794 (+591713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.389173 1919 rpcz_store.cc:276] 0626 08:24:05.817895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.818123 (+ 228us) service_pool.cc:225] Handling call
0626 08:24:05.818553 (+ 430us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:06.027896 (+209343us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:06.027944 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:06.028676 (+ 732us) log.cc:844] Serialized 52 byte log entry
0626 08:24:06.202777 (+174101us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:06.226629 (+ 23852us) log.cc:844] Serialized 14 byte log entry
0626 08:24:06.226850 (+ 221us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:06.226905 (+ 55us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:07.319496 (+1092591us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b1800363880. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:09.551683 (+2232187us) spinlock_profiling.cc:243] Waited 7.02 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:10.346469 (+794786us) raft_consensus.cc:1683] finished
0626 08:24:10.346491 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.388656 (+ 42165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11284,"prepare.queue_time_us":23782,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":150890,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":8797,"spinlock_wait_cycles":25361536,"thread_start_us":277793,"threads_started":2,"wal-append.queue_time_us":168147}
I20250626 08:24:10.420785 1994 raft_consensus.cc:2391] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:10.421471 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2173) took 1003 ms. Trace:
I20250626 08:24:10.421888 1994 rpcz_store.cc:276] 0626 08:24:09.418449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.802451 (+384002us) service_pool.cc:225] Handling call
0626 08:24:10.421386 (+618935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.423134 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:10.423980 1994 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:10.425074 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:10.395684 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:09.939874 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 149976us
W20250626 08:24:09.843274 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 141435us
I20250626 08:24:10.427644 2582 rpcz_store.cc:276] 0626 08:24:06.312267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.385272 (+ 73005us) service_pool.cc:225] Handling call
0626 08:24:09.506720 (+3121448us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:09.506937 (+ 217us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:09.794163 (+287226us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.794357 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24270080}
I20250626 08:24:10.428591 1812 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:09.827346 4849 log.cc:927] Time spent T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
W20250626 08:24:09.940353 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.429720 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2174) took 1007 ms. Trace:
I20250626 08:24:10.006996 4735 raft_consensus.cc:3053] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:09.815956 4539 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.806933 4715 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:09.859323 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.021373 4496 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.458835 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:10.464864 1995 raft_consensus.cc:2374] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7. Re-sending same reply.
I20250626 08:24:10.416185 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.806387 4749 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:10.017498 4609 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.519s user 0.011s sys 0.002s
I20250626 08:24:09.997817 4877 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:10.005513 4859 log.cc:927] Time spent T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.281s user 0.000s sys 0.000s
W20250626 08:24:10.438618 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.416890 4774 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.808458 3988 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.445703 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:09.939415 4846 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20250626 08:24:09.980157 4625 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250626 08:24:09.997992 4632 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.739s user 0.005s sys 0.007s
I20250626 08:24:10.026438 4873 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:09.972421 4629 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:10.394212 1999 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:24:09.941159 4852 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.456028 1997 rpcz_store.cc:276] 0626 08:24:09.422344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.571686 (+149342us) service_pool.cc:225] Handling call
0626 08:24:10.429632 (+857946us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:10.465006 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.048s sys 0.064s
I20250626 08:24:10.385876 4875 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:09.939500 4667 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:09.649176 2579 rpcz_store.cc:276] 0626 08:24:06.730324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.840162 (+109838us) service_pool.cc:225] Handling call
0626 08:24:06.840541 (+ 379us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:06.857883 (+ 17342us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:06.859774 (+ 1891us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:07.060737 (+200963us) log.cc:844] Serialized 52 byte log entry
0626 08:24:08.538350 (+1477613us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:08.543786 (+ 5436us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:08.546367 (+ 2581us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:09.562493 (+1016126us) raft_consensus.cc:1683] finished
0626 08:24:09.562516 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:09.562677 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3186,"raft.queue_time_us":17679,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":1757,"spinlock_wait_cycles":1466240,"thread_start_us":1339237,"threads_started":1,"wal-append.queue_time_us":1533883}
I20250626 08:24:10.632846 4774 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:10.061041 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 348477us
I20250626 08:24:10.633610 4803 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.590409 4778 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.634387 4778 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.627509 4852 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:10.465557 1995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2142) took 3406 ms (client timeout 1938 ms). Trace:
I20250626 08:24:10.465710 1996 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:10.636406 4778 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.637198 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:10.637420 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2026) took 3645 ms. Trace:
I20250626 08:24:10.614446 4397 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.637769 1925 rpcz_store.cc:276] 0626 08:24:06.991661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.992013 (+ 352us) service_pool.cc:225] Handling call
0626 08:24:06.992306 (+ 293us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.456507 (+1464201us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.637190 (+2180683us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001a9854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:10.637371 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3360131968}
I20250626 08:24:10.638257 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:09.981014 4839 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.966931 4838 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.638545 4397 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.638939 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:10.639353 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 196526us
I20250626 08:24:10.639539 1925 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:10.640091 4797 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:10.640297 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.640527 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:10.640708 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.437634 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:10.017205 4813 consensus_meta.cc:287] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.621s user 0.007s sys 0.007s
I20250626 08:24:09.928735 4798 consensus_queue.cc:1035] T 26cfef3699a141f8b7686ae2b75a3beb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.540s
W20250626 08:24:09.827734 4857 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.330s user 0.000s sys 0.001s
I20250626 08:24:09.817157 4856 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:09.813498 4863 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.448992 4711 raft_consensus.cc:1025] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: attempt to promote peer be56082cacb54e9aa8f1a9ee91e70d76: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250626 08:24:10.607708 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:10.432682 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.000s sys 0.014s
W20250626 08:24:10.654852 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.000s sys 0.014s
W20250626 08:24:10.455338 4599 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.992s user 0.011s sys 0.000s
I20250626 08:24:10.609421 4887 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.611462 4731 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:09.974943 4733 consensus_queue.cc:260] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.661146 4733 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:10.661410 4728 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:10.661772 4599 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:09.967330 4664 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.624301 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:09.997627 4486 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.466873 4841 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:09.980692 4842 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:10.593336 4581 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.008s sys 0.025s
I20250626 08:24:10.602823 4625 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:10.629983 4885 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.631088 3988 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.631358 4624 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.439517 4700 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.459412 4735 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:10.671650 4581 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.008s sys 0.026s
I20250626 08:24:10.671845 4625 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:10.632354 4772 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.678974 4735 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.632504 4667 raft_consensus.cc:695] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:10.680104 4735 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:10.680639 4667 consensus_queue.cc:237] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.639205 4838 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.685268 4746 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:10.685959 4838 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.640551 4797 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.688251 4797 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:10.642501 4712 consensus_meta.cc:287] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 2.092s user 0.012s sys 0.002s
I20250626 08:24:10.689247 4865 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.614262 4496 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.690752 4496 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.691375 4863 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:10.636253 1995 rpcz_store.cc:269] 0626 08:24:07.059092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.059256 (+ 164us) service_pool.cc:225] Handling call
0626 08:24:10.458663 (+3399407us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:10.465398 (+ 6735us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35298688}
I20250626 08:24:10.692808 4856 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.671701 4652 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:10.643028 4558 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.831s user 0.009s sys 0.000s
I20250626 08:24:10.694351 4652 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.694612 4558 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.617765 4873 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.696398 4652 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.606668 4840 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.606797 4629 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:10.614122 4871 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.625321 4853 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.628751 4189 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:09.811926 4719 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.637s
I20250626 08:24:10.689787 4865 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.703135 4853 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:10.645963 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.705183 4853 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.705889 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:10.705673 4871 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:10.613430 4869 log.cc:927] Time spent T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.278s user 0.000s sys 0.000s
I20250626 08:24:10.707635 4603 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.707908 4871 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:10.622295 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.684s user 0.053s sys 0.069s
I20250626 08:24:10.535665 4882 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:10.675100 4885 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.697978 4599 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.640774 4397 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.663476 4643 raft_consensus.cc:3053] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:10.446465 4681 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.608209 4877 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.677943 4700 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.631569 4875 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:24:10.718039 4643 raft_consensus.cc:738] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:10.722576 4643 consensus_queue.cc:260] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.723366 4875 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.723538 4643 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:10.725226 4877 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.727120 4681 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.713680 4882 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.729502 4882 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.650908 4813 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.659909 4887 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.458249 4848 consensus_queue.cc:1035] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.171s
I20250626 08:24:10.700760 4840 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.537106 4884 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:10.712520 4719 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:10.715179 4852 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.737711 4884 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.738696 4873 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.739292 4887 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.737434 4719 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:10.740200 4719 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:10.740387 4813 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.740409 4884 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.738781 4885 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.768070 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:24:10.457397 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.460911 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.602514 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:10.774055 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:10.775171 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.775429 1920 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250626 08:24:10.775607 1841 raft_consensus.cc:2408] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:10.613853 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.652761 1996 raft_consensus.cc:2408] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20250626 08:24:10.650485 2000 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.588s user 0.006s sys 0.027s
I20250626 08:24:10.469050 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:10.645282 1919 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:10.778872 1850 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:10.779121 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2050) took 1063 ms. Trace:
W20250626 08:24:10.575126 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 140982us
I20250626 08:24:10.560154 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.779757 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2145) took 3717 ms. Trace:
I20250626 08:24:10.589941 1844 raft_consensus.cc:3053] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:10.781572 1844 raft_consensus.cc:738] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:10.780352 2000 rpcz_store.cc:276] 0626 08:24:07.062464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.062610 (+ 146us) service_pool.cc:225] Handling call
0626 08:24:09.564590 (+2501980us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:09.564778 (+ 188us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:09.565141 (+ 363us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:09.565171 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:09.565585 (+ 414us) log.cc:844] Serialized 26 byte log entry
0626 08:24:09.796831 (+231246us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:10.778048 (+981217us) log.cc:844] Serialized 14 byte log entry
0626 08:24:10.778763 (+ 715us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.778947 (+ 184us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:10.779035 (+ 88us) raft_consensus.cc:1683] finished
0626 08:24:10.779173 (+ 138us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.779437 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16121,"prepare.queue_time_us":392003,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":26040,"raft.queue_time_us":5278,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":15470,"spinlock_wait_cycles":21057408,"thread_start_us":945047,"threads_started":2,"wal-append.queue_time_us":422572}
I20250626 08:24:10.612836 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2024) took 3621 ms. Trace:
I20250626 08:24:10.783546 1844 consensus_queue.cc:260] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:10.784370 1927 rpcz_store.cc:276] 0626 08:24:06.991032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.992550 (+ 1518us) service_pool.cc:225] Handling call
0626 08:24:06.992871 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:07.934302 (+941431us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.612579 (+2678277us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54002e04d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:10.612770 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2033106688}
I20250626 08:24:10.785393 2000 raft_consensus.cc:1273] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:10.641144 1925 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:10.786278 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2154) took 3708 ms. Trace:
I20250626 08:24:10.699097 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2147) took 3635 ms. Trace:
I20250626 08:24:10.787901 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:10.655393 1967 connection.cc:232] RPC call timeout handler was delayed by -1.9481862969996655s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20250626 08:24:10.789247 1925 raft_consensus.cc:2408] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:24:10.666186 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2150) took 3599 ms. Trace:
W20250626 08:24:10.589566 1892 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.022s sys 0.065s
I20250626 08:24:10.474977 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:10.555307 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.465157 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.697790 1994 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 9: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:10.674999 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1742) took 4258 ms. Trace:
I20250626 08:24:10.794391 1995 raft_consensus.cc:1238] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:10.698390 4308 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:10.692808 4496 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:10.696412 4558 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.700443 4629 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:10.708181 4603 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.813808 4581 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:10.814100 4896 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250626 08:24:10.813987 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:10.815053 4581 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:10.816520 4603 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.816756 1993 rpcz_store.cc:276] 0626 08:24:07.066300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.545760 (+1479460us) service_pool.cc:225] Handling call
0626 08:24:08.563530 (+ 17770us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:08.570595 (+ 7065us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:08.574087 (+ 3492us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:08.595172 (+ 21085us) log.cc:844] Serialized 26 byte log entry
0626 08:24:09.480410 (+885238us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:09.480449 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.480525 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:10.665879 (+1185354us) raft_consensus.cc:1683] finished
0626 08:24:10.665937 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.666108 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1151,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6393,"thread_start_us":768825,"threads_started":1,"wal-append.queue_time_us":1045295}
I20250626 08:24:10.818687 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:10.787536 2001 rpcz_store.cc:276] 0626 08:24:07.064050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.372690 (+1308640us) service_pool.cc:225] Handling call
0626 08:24:08.409509 (+ 36819us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:08.410013 (+ 504us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:08.410057 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:08.410886 (+ 829us) log.cc:844] Serialized 26 byte log entry
0626 08:24:09.251214 (+840328us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:09.251271 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.251361 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:10.698790 (+1447429us) raft_consensus.cc:1683] finished
0626 08:24:10.698820 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.699017 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5572,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":2367,"thread_start_us":806015,"threads_started":1,"wal-append.queue_time_us":1025289}
I20250626 08:24:10.785560 1844 raft_consensus.cc:3058] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:10.777139 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:10.794658 1892 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.022s sys 0.071s
I20250626 08:24:10.786808 2000 rpcz_store.cc:276] 0626 08:24:07.079066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.784101 (+3705035us) service_pool.cc:225] Handling call
0626 08:24:10.784782 (+ 681us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.785966 (+ 1184us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.786179 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.779724 1919 rpcz_store.cc:276] 0626 08:24:09.715455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.437205 (+721750us) service_pool.cc:225] Handling call
0626 08:24:10.778771 (+341566us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.786819 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:10.820071 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:10.795653 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:10.830438 1999 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:10.831799 1993 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:10.835440 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:10.832738 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.834797 1925 raft_consensus.cc:2408] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:10.835767 1926 raft_consensus.cc:2408] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 10 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250626 08:24:10.837152 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2044) took 2165 ms. Trace:
I20250626 08:24:10.816193 1998 rpcz_store.cc:276] 0626 08:24:06.414030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.414520 (+ 490us) service_pool.cc:225] Handling call
0626 08:24:06.414929 (+ 409us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:06.556325 (+141396us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:07.870169 (+1313844us) log.cc:844] Serialized 14 byte log entry
0626 08:24:09.253656 (+1383487us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:09.254285 (+ 629us) log.cc:844] Serialized 26 byte log entry
0626 08:24:09.254444 (+ 159us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:09.695512 (+441068us) log.cc:844] Serialized 14 byte log entry
0626 08:24:09.696677 (+ 1165us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:09.697591 (+ 914us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:10.671787 (+974196us) raft_consensus.cc:1683] finished
0626 08:24:10.671817 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.671986 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":138162,"prepare.queue_time_us":655498,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":75,"raft.queue_time_us":102083,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":22631,"thread_start_us":3228433,"threads_started":4,"wal-append.queue_time_us":1570426}
I20250626 08:24:10.837633 1926 rpcz_store.cc:276] 0626 08:24:08.672245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.466560 (+1794315us) service_pool.cc:225] Handling call
0626 08:24:10.837036 (+370476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.834108 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.840837 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:10.841114 1847 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:24:10.841296 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:10.842504 1926 raft_consensus.cc:2408] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:10.790799 1967 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:10.843571 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:10.843690 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:10.844606 1926 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:10.844825 1925 raft_consensus.cc:2408] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:10.845052 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1887) took 1093 ms. Trace:
I20250626 08:24:10.845443 1926 rpcz_store.cc:276] 0626 08:24:09.752005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.843143 (+1091138us) service_pool.cc:225] Handling call
0626 08:24:10.844979 (+ 1836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.848838 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:10.873318 2582 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:10.875639 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:10.875691 1924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2041) took 2267 ms (client timeout 2121 ms). Trace:
I20250626 08:24:10.850557 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.852236 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:10.854548 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1886) took 1103 ms. Trace:
I20250626 08:24:10.871623 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.776093 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.844864 1919 raft_consensus.cc:1273] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20250626 08:24:10.863238 4581 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:10.864590 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 546) took 3293 ms. Trace:
W20250626 08:24:10.866923 1928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 545) took 3329 ms (client timeout 2035 ms). Trace:
I20250626 08:24:10.867063 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:60106 (request call id 992) took 5316 ms. Trace:
I20250626 08:24:10.870364 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:10.876408 1924 rpcz_store.cc:269] 0626 08:24:08.608392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.587487 (+979095us) service_pool.cc:225] Handling call
0626 08:24:10.875128 (+1287641us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001a9d54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:10.875385 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2581536640}
I20250626 08:24:10.803114 4486 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:10.801398 2580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 670) took 1545 ms. Trace:
I20250626 08:24:10.878540 1920 rpcz_store.cc:276] 0626 08:24:09.751631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.445345 (+693714us) service_pool.cc:225] Handling call
0626 08:24:10.854328 (+408983us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54003124d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:10.854494 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":391958400}
I20250626 08:24:10.877056 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.883424 1845 rpcz_store.cc:276] 0626 08:24:05.550857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:05.551081 (+ 224us) service_pool.cc:225] Handling call
0626 08:24:10.857394 (+5306313us) spinlock_profiling.cc:243] Waited 4.26 s on lock 0x7b540032ebd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003a70 00007f4eb9087b1a 00007f4ebdfd509e 00007f4eb767f8c7 00007f4eb767f852 00007f4eb767f781 00007f4eb767f6fc 00007f4eb767e9b2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:10.866818 (+ 9424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9789190528}
I20250626 08:24:10.892256 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.893227 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.894117 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.895077 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.895176 1846 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:10.896849 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.897334 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.898063 1845 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:10.906999 1927 raft_consensus.cc:2391] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:10.908215 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.908901 1848 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
W20250626 08:24:10.910092 4671 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 3.540s user 0.022s sys 0.048s
W20250626 08:24:10.910665 4671 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 3.565s user 0.023s sys 0.049s
I20250626 08:24:10.910377 4486 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:10.882884 1928 rpcz_store.cc:269] 0626 08:24:07.537160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:07.537379 (+ 219us) service_pool.cc:225] Handling call
0626 08:24:10.866448 (+3329069us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54001a8e54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:10.866703 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3971926272}
I20250626 08:24:10.913220 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:10.901796 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2032) took 3307 ms. Trace:
I20250626 08:24:10.908906 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2034) took 3328 ms. Trace:
W20250626 08:24:11.975713 1850 consensus_meta.cc:287] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.196s user 0.026s sys 0.025s
I20250626 08:24:11.618435 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1830) took 1158 ms. Trace:
I20250626 08:24:11.979110 1850 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:11.027485 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:11.019660 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.013s sys 0.008s
I20250626 08:24:11.024150 1847 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:10.995549 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:11.135257 1998 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
W20250626 08:24:11.897537 1920 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.012s sys 0.027s
W20250626 08:24:11.105960 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:11.984673 1920 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s user 0.016s sys 0.028s
I20250626 08:24:10.991349 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:11.191566 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2027) took 4199 ms. Trace:
W20250626 08:24:11.409247 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.972824 1841 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:10.991734 1925 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250626 08:24:10.913677 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 548) took 3309 ms. Trace:
I20250626 08:24:10.918704 1926 rpcz_store.cc:276] 0626 08:24:07.579678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.872123 (+3292445us) service_pool.cc:225] Handling call
0626 08:24:10.872704 (+ 581us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.876501 (+ 3797us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.877223 (+ 722us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:10.886875 (+ 9652us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:10.886937 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.887025 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.887171 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:10.882782 1923 rpcz_store.cc:276] 0626 08:24:07.570797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:08.377374 (+806577us) service_pool.cc:225] Handling call
0626 08:24:08.377636 (+ 262us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.428366 (+ 50730us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:08.428398 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:08.428449 (+ 51us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:08.428483 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.864122 (+2435639us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b54003683d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:10.864195 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.864416 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116074880}
I20250626 08:24:10.943838 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:11.979663 1927 rpcz_store.cc:276] 0626 08:24:07.580367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.907929 (+3327562us) service_pool.cc:225] Handling call
0626 08:24:10.908334 (+ 405us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.908565 (+ 231us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.908591 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:10.908633 (+ 42us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:10.908665 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.908719 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.908817 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:11.987216 1919 rpcz_store.cc:276] 0626 08:24:07.578440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.843936 (+3265496us) service_pool.cc:225] Handling call
0626 08:24:10.844438 (+ 502us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.879325 (+ 34887us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.887567 (+ 8242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:11.979225 1844 rpcz_store.cc:276] 0626 08:24:10.460226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.460332 (+ 106us) service_pool.cc:225] Handling call
0626 08:24:10.555765 (+ 95433us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:11.392080 (+836315us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:11.421196 (+ 29116us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:11.421270 (+ 74us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:11.591445 (+170175us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.591544 (+ 99us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:11.618314 (+ 26770us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47230,"raft.queue_time_us":136032,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":149133}
I20250626 08:24:11.982210 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:11.055619 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2038) took 2491 ms. Trace:
I20250626 08:24:11.131621 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 414) took 1323 ms. Trace:
I20250626 08:24:11.991420 1928 rpcz_store.cc:276] 0626 08:24:08.564376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.943658 (+2379282us) service_pool.cc:225] Handling call
0626 08:24:11.054771 (+111113us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:11.055165 (+ 394us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:11.055203 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:11.055280 (+ 77us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:11.055323 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.055394 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:11.055522 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:11.992978 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.932s user 0.024s sys 0.041s
I20250626 08:24:11.993355 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1842) took 1136 ms. Trace:
I20250626 08:24:11.994055 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1848) took 1139 ms. Trace:
I20250626 08:24:11.048771 4671 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:11.995607 4629 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.080s user 0.013s sys 0.003s
I20250626 08:24:11.105386 1965 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:11.998521 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2161) took 4365 ms. Trace:
I20250626 08:24:11.999328 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 638) took 1058 ms. Trace:
I20250626 08:24:11.993677 1847 rpcz_store.cc:276] 0626 08:24:10.845624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.850175 (+ 4551us) service_pool.cc:225] Handling call
0626 08:24:11.981819 (+1131644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:11.986456 1918 rpcz_store.cc:276] 0626 08:24:06.991991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.994043 (+ 2052us) service_pool.cc:225] Handling call
0626 08:24:06.994382 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:08.000895 (+1006513us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.190827 (+3189932us) spinlock_profiling.cc:243] Waited 971 ms on lock 0x7b54003133d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:11.191352 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2234373888}
I20250626 08:24:11.999119 1998 rpcz_store.cc:276] 0626 08:24:07.626102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:11.125212 (+3499110us) service_pool.cc:225] Handling call
0626 08:24:11.125653 (+ 441us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:11.990835 (+865182us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.991144 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:11.994329 1841 rpcz_store.cc:276] 0626 08:24:10.848291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.871129 (+ 22838us) service_pool.cc:225] Handling call
0626 08:24:11.988081 (+1116952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.001608 4671 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:12.002205 4746 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.102s user 0.016s sys 0.005s
I20250626 08:24:12.009137 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2033) took 4424 ms. Trace:
I20250626 08:24:12.009443 1920 rpcz_store.cc:276] 0626 08:24:07.580066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.890681 (+3310615us) service_pool.cc:225] Handling call
0626 08:24:10.905241 (+ 14560us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.905569 (+ 328us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:24:10.991658 (+ 86089us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.003750 (+1012092us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.003806 (+ 56us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:12.003850 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.003926 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.004044 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1028184,"threads_started":2}
I20250626 08:24:11.992475 1996 rpcz_store.cc:276] 0626 08:24:09.807735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.638044 (+830309us) service_pool.cc:225] Handling call
0626 08:24:11.131182 (+493138us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b54002035d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:11.131401 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31208192}
I20250626 08:24:11.975742 4486 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.020153 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2047) took 2483 ms. Trace:
W20250626 08:24:12.022619 1849 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.814s user 0.007s sys 0.015s
W20250626 08:24:12.023209 1849 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.064s user 0.007s sys 0.015s
W20250626 08:24:12.024256 2584 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.004s sys 0.031s
W20250626 08:24:12.024807 2584 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.001s user 0.004s sys 0.031s
I20250626 08:24:12.020860 1926 rpcz_store.cc:276] 0626 08:24:09.536239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.018526 (+2482287us) service_pool.cc:225] Handling call
0626 08:24:12.019116 (+ 590us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.019505 (+ 389us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.019558 (+ 53us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.019656 (+ 98us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:12.019714 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.019823 (+ 109us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.020037 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:12.027563 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1833) took 1559 ms (client timeout 1713 ms). Trace:
I20250626 08:24:11.999986 2582 rpcz_store.cc:276] 0626 08:24:10.934048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.936397 (+ 2349us) service_pool.cc:225] Handling call
0626 08:24:11.992550 (+1056153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.008496 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1854) took 1096 ms. Trace:
W20250626 08:24:12.015520 4625 consensus_meta.cc:287] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.266s user 0.010s sys 0.008s
I20250626 08:24:12.001650 4629 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.033077 4865 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:11.195024 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.45387e+06us
W20250626 08:24:11.106457 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 316172us
W20250626 08:24:12.036350 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:10.902329 2579 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
W20250626 08:24:12.029850 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.028628 1848 rpcz_store.cc:276] 0626 08:24:10.911932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.919506 (+ 7574us) service_pool.cc:225] Handling call
0626 08:24:12.008375 (+1088869us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.031327 1846 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 9: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:12.044713 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:11.748262 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 395485us
I20250626 08:24:12.033643 4629 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.036815 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:11.129977 4832 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:12.043279 4486 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:12.048615 4746 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:12.048552 4832 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.004887 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 549) took 4418 ms. Trace:
W20250626 08:24:12.041431 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.042544 4625 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.044792 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1737) took 1131 ms. Trace:
I20250626 08:24:12.031683 4634 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:12.045593 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.047363 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.049181 4746 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.055518 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.056136 4746 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.056433 1847 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.057713 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:12.055186 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2040) took 3447 ms. Trace:
I20250626 08:24:12.058075 4865 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:24:12.058877 4643 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:12.059300 4643 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:12.050734 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:11.055871 4667 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:12.050685 1925 rpcz_store.cc:276] 0626 08:24:07.579312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.845420 (+3266108us) service_pool.cc:225] Handling call
0626 08:24:10.884637 (+ 39217us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:11.997092 (+1112455us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.997341 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.059669 1814 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:12.053967 4840 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.048647 4486 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.055183 4634 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:12.062017 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.052500 1846 rpcz_store.cc:276] 0626 08:24:10.913144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.975434 (+ 62290us) service_pool.cc:225] Handling call
0626 08:24:12.044743 (+1069309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.058391 1928 rpcz_store.cc:276] 0626 08:24:08.608096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:11.993048 (+3384952us) service_pool.cc:225] Handling call
0626 08:24:12.054449 (+ 61401us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.054793 (+ 344us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.054824 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.054894 (+ 70us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:12.054938 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.055020 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.055136 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:12.028085 1850 rpcz_store.cc:269] 0626 08:24:10.468613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.468746 (+ 133us) service_pool.cc:225] Handling call
0626 08:24:12.010307 (+1541561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.066890 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.060426 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1829) took 1600 ms. Trace:
I20250626 08:24:12.070099 1845 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:12.080374 1923 raft_consensus.cc:2408] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:12.068262 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.051913 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 634) took 1204 ms. Trace:
I20250626 08:24:12.076763 1842 rpcz_store.cc:276] 0626 08:24:10.459855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.459962 (+ 107us) service_pool.cc:225] Handling call
0626 08:24:10.535867 (+ 75905us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.600855 (+ 64988us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.600882 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:10.600927 (+ 45us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:10.600956 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.060079 (+1459123us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b54002f2bd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:12.060136 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.060276 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148909952}
W20250626 08:24:12.081343 1845 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1853) took 1193 ms (client timeout 1563 ms). Trace:
I20250626 08:24:12.085451 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1855) took 1173 ms. Trace:
I20250626 08:24:12.060791 4671 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:12.086045 1845 rpcz_store.cc:269] 0626 08:24:10.901713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.918653 (+ 16940us) service_pool.cc:225] Handling call
0626 08:24:12.081274 (+1162621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.059707 4643 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:12.085832 1843 rpcz_store.cc:276] 0626 08:24:10.912350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.919423 (+ 7073us) service_pool.cc:225] Handling call
0626 08:24:12.066690 (+1147267us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800000f98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:12.085388 (+ 18698us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6453120}
I20250626 08:24:12.090188 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1889) took 2285 ms. Trace:
I20250626 08:24:12.090656 1927 rpcz_store.cc:276] 0626 08:24:09.804807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.085971 (+2281164us) service_pool.cc:225] Handling call
0626 08:24:12.089313 (+ 3342us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.089792 (+ 479us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.089820 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.089884 (+ 64us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.089967 (+ 83us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.090032 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.090134 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.093123 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 636) took 1244 ms. Trace:
I20250626 08:24:12.093551 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2152) took 5023 ms. Trace:
I20250626 08:24:12.094355 4832 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.093948 1997 rpcz_store.cc:276] 0626 08:24:07.077177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.660180 (+3583003us) service_pool.cc:225] Handling call
0626 08:24:10.660579 (+ 399us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:10.795506 (+134927us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.795570 (+ 64us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:10.818309 (+ 22739us) log.cc:844] Serialized 78 byte log entry
0626 08:24:10.890810 (+ 72501us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:10.891786 (+ 976us) log.cc:844] Serialized 14 byte log entry
0626 08:24:11.978497 (+1086711us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.008016 (+ 29519us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.093166 (+ 85150us) raft_consensus.cc:1683] finished
0626 08:24:12.093184 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.093350 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10221,"prepare.queue_time_us":373,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":56338,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":42515,"thread_start_us":206078,"threads_started":2,"wal-append.queue_time_us":1271555}
I20250626 08:24:12.096230 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.096743 4719 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.356s user 0.014s sys 0.004s
I20250626 08:24:12.066936 4895 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:10.890293 2580 rpcz_store.cc:276] 0626 08:24:09.256050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:09.257796 (+ 1746us) service_pool.cc:225] Handling call
0626 08:24:10.799562 (+1541766us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:10.800124 (+ 562us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.800694 (+ 570us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.800761 (+ 67us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:10.800850 (+ 89us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:10.800927 (+ 77us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:10.801029 (+ 102us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:10.801200 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191435008}
I20250626 08:24:12.097098 1997 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.102309 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.093534 2584 rpcz_store.cc:276] 0626 08:24:10.848285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.849452 (+ 1167us) service_pool.cc:225] Handling call
0626 08:24:10.849887 (+ 435us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:10.850722 (+ 835us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:10.990584 (+139862us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.025674 (+1035090us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:12.046453 (+ 20779us) log.cc:844] Serialized 26 byte log entry
0626 08:24:12.046558 (+ 105us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:12.046732 (+ 174us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.046833 (+ 101us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.046879 (+ 46us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.092899 (+ 46020us) raft_consensus.cc:1683] finished
0626 08:24:12.092926 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.093071 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":967,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":367,"thread_start_us":1124600,"threads_started":2,"wal-append.queue_time_us":1066811}
W20250626 08:24:12.109737 1966 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.107721 1816 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.084929 2579 rpcz_store.cc:276] 0626 08:24:10.847346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.851879 (+ 4533us) service_pool.cc:225] Handling call
0626 08:24:12.047032 (+1195153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.112866 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1885) took 2367 ms. Trace:
I20250626 08:24:12.106179 4840 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:12.113193 1925 rpcz_store.cc:276] 0626 08:24:09.745312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.069785 (+2324473us) service_pool.cc:225] Handling call
0626 08:24:12.101577 (+ 31792us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.109804 (+ 8227us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.109849 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.112677 (+ 2828us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.112706 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.112745 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.112826 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.118482 2002 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:24:12.117131 4840 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.126665 4840 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:12.100356 4719 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.124909 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1895) took 1717 ms. Trace:
I20250626 08:24:12.119796 1996 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:12.130451 1923 rpcz_store.cc:276] 0626 08:24:10.407798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.100043 (+1692245us) service_pool.cc:225] Handling call
0626 08:24:12.112718 (+ 12675us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.116964 (+ 4246us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.117000 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.124495 (+ 7495us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.124559 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.124658 (+ 99us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.124836 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.132495 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:12.120666 4662 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.370s user 0.019s sys 0.038s
I20250626 08:24:12.138540 1925 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20250626 08:24:12.114434 4728 consensus_meta.cc:287] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.379s user 0.017s sys 0.001s
I20250626 08:24:12.140224 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2164) took 4491 ms. Trace:
I20250626 08:24:12.140812 1998 rpcz_store.cc:276] 0626 08:24:07.645177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.041066 (+4395889us) service_pool.cc:225] Handling call
0626 08:24:12.079976 (+ 38910us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.119496 (+ 39520us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.119514 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.128577 (+ 9063us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.136163 (+ 7586us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.136254 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.136414 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.142527 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2043) took 3531 ms. Trace:
I20250626 08:24:12.143414 1918 rpcz_store.cc:276] 0626 08:24:08.608934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.010660 (+3401726us) service_pool.cc:225] Handling call
0626 08:24:12.086413 (+ 75753us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.136610 (+ 50197us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.136652 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.139943 (+ 3291us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:12.139978 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.140031 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.140118 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":644352}
I20250626 08:24:12.134955 4865 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:12.145550 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2046) took 3429 ms. Trace:
I20250626 08:24:12.145860 1920 rpcz_store.cc:276] 0626 08:24:08.715890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.010275 (+3294385us) service_pool.cc:225] Handling call
0626 08:24:12.068033 (+ 57758us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.138029 (+ 69996us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.138072 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.141863 (+ 3791us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.145361 (+ 3498us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.145426 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.145508 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:12.142319 4662 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.467s user 0.022s sys 0.041s
I20250626 08:24:12.149758 1923 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:24:12.157529 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.166483 1997 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.167143 1919 raft_consensus.cc:1238] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:12.117236 4895 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.169476 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2039) took 3565 ms. Trace:
I20250626 08:24:12.171016 1919 rpcz_store.cc:276] 0626 08:24:08.603935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:11.996559 (+3392624us) service_pool.cc:225] Handling call
0626 08:24:12.165054 (+168495us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:12.165320 (+ 266us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.168718 (+ 3398us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.169384 (+ 666us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3421440}
I20250626 08:24:12.169447 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.180933 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1888) took 2428 ms. Trace:
I20250626 08:24:12.187330 1841 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
I20250626 08:24:12.182735 1928 rpcz_store.cc:276] 0626 08:24:09.752325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.081669 (+2329344us) service_pool.cc:225] Handling call
0626 08:24:12.113062 (+ 31393us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.169480 (+ 56418us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.169516 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.173271 (+ 3755us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:12.180719 (+ 7448us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.180776 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.180880 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4156,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":5877}
W20250626 08:24:12.190029 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 86574.6us
I20250626 08:24:12.198341 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2035) took 3644 ms. Trace:
I20250626 08:24:12.198659 1924 rpcz_store.cc:276] 0626 08:24:08.554402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.934816 (+2380414us) service_pool.cc:225] Handling call
0626 08:24:12.164221 (+1229405us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:12.168458 (+ 4237us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.191840 (+ 23382us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.191885 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.191952 (+ 67us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:12.198157 (+ 6205us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.198217 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.198298 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26576768}
I20250626 08:24:12.199633 1996 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:12.197109 1998 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:12.192866 4486 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.205310 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2019) took 5825 ms. Trace:
I20250626 08:24:12.207541 1921 rpcz_store.cc:276] 0626 08:24:06.369085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.369239 (+ 154us) service_pool.cc:225] Handling call
0626 08:24:06.369538 (+ 299us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:08.545239 (+2175701us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.169286 (+3624047us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54001e1754. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:12.194406 (+ 25120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4993775872}
I20250626 08:24:12.209028 4719 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.218757 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.215991 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 929659us
I20250626 08:24:12.212370 4486 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:12.222468 1999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 402) took 2770 ms (client timeout 2128 ms). Trace:
W20250626 08:24:12.226050 1999 rpcz_store.cc:269] 0626 08:24:09.450554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.623928 (+1173374us) service_pool.cc:225] Handling call
0626 08:24:12.216642 (+1592714us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54001662d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:12.220478 (+ 3836us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368873344}
I20250626 08:24:12.233150 4634 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.224196 4865 consensus_queue.cc:1046] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250626 08:24:12.246052 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.239073 1968 connection.cc:232] RPC call timeout handler was delayed by -1.0126824379999562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
I20250626 08:24:12.252645 1996 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.254033 4486 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.256129 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.256913 1996 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:12.257814 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.258044 4486 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.258247 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.258437 1996 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.259871 1847 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.261042 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1738) took 1288 ms. Trace:
I20250626 08:24:12.261407 1847 rpcz_store.cc:276] 0626 08:24:10.976285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.044748 (+1068463us) service_pool.cc:225] Handling call
0626 08:24:12.260973 (+216225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.262665 4643 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:12.264979 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 398) took 5556 ms (client timeout 3220 ms). Trace:
I20250626 08:24:12.267825 1844 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.267907 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.265738 2002 rpcz_store.cc:269] 0626 08:24:06.708736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:06.708883 (+ 147us) service_pool.cc:225] Handling call
0626 08:24:12.264741 (+5555858us) spinlock_profiling.cc:243] Waited 5.38 s on lock 0x7b5400078ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:12.264880 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12373702912}
W20250626 08:24:12.261001 4906 log.cc:927] Time spent T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20250626 08:24:12.275563 1998 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
I20250626 08:24:12.247143 4746 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.267129 4634 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:12.279345 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.281536 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.279289 4671 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.283466 1999 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:12.282290 2002 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 3.
I20250626 08:24:12.282224 4634 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.286295 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.286909 1844 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.283468 4671 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:12.290176 4902 log.cc:927] Time spent T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250626 08:24:12.290463 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.293653 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.290526 4634 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.290732 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2153) took 5213 ms. Trace:
I20250626 08:24:12.296765 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.303445 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.303889 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1840) took 1530 ms. Trace:
I20250626 08:24:12.305158 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.304344 1849 rpcz_store.cc:276] 0626 08:24:10.773519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.780997 (+ 7478us) service_pool.cc:225] Handling call
0626 08:24:10.781561 (+ 564us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:10.782454 (+ 893us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.782515 (+ 61us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:10.783405 (+ 890us) log.cc:844] Serialized 26 byte log entry
0626 08:24:10.921445 (+138040us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:12.023741 (+1102296us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.026769 (+ 3028us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.026874 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.303673 (+276799us) raft_consensus.cc:1683] finished
0626 08:24:12.303696 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.303831 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37628,"prepare.queue_time_us":1144831,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":44158,"raft.run_cpu_time_us":694,"raft.run_wall_time_us":170619,"thread_start_us":1202020,"threads_started":2,"wal-append.queue_time_us":1336430}
I20250626 08:24:12.295680 1995 rpcz_store.cc:276] 0626 08:24:07.077622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.693215 (+3615593us) service_pool.cc:225] Handling call
0626 08:24:10.693553 (+ 338us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:10.837981 (+144428us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.278294 (+1440313us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540007a8d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:12.290658 (+ 12364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230294784}
I20250626 08:24:12.307360 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.282953 4895 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.313534 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.285790 4643 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:12.315336 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 1.34118e+06us
W20250626 08:24:12.315474 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 103942us
I20250626 08:24:12.313566 4840 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.321903 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.322115 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.322178 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.322747 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.323618 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.327836 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.330591 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.330818 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.331511 1924 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:12.331579 2584 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
W20250626 08:24:12.331876 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2172) took 2928 ms (client timeout 2121 ms). Trace:
W20250626 08:24:12.332273 1994 rpcz_store.cc:269] 0626 08:24:09.403445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.424757 (+1021312us) service_pool.cc:225] Handling call
0626 08:24:12.331541 (+1906784us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54000794d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:12.331693 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":605060352}
I20250626 08:24:12.332665 1999 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.333107 4629 raft_consensus.cc:3053] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:12.333498 4629 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:12.334121 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.335005 1928 raft_consensus.cc:2391] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:12.334589 4629 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.336796 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.339300 1995 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.339385 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1739) took 1287 ms. Trace:
I20250626 08:24:12.340036 1841 rpcz_store.cc:276] 0626 08:24:11.088707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.010594 (+921887us) service_pool.cc:225] Handling call
0626 08:24:12.339177 (+328583us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b5400180bd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:12.339311 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211911552}
W20250626 08:24:12.327682 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 51787.8us
I20250626 08:24:12.334990 1921 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:12.333374 2580 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:12.349848 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:12.355501 2550 leader_election.cc:336] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 1.612s (SENT)
I20250626 08:24:12.355062 1845 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
I20250626 08:24:12.357196 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.357342 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.359043 2550 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:12.361960 4486 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:12.362251 2551 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.362581 4486 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.363253 4719 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:12.363759 4719 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.364200 4719 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.364626 4486 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:12.364871 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 576501us
I20250626 08:24:12.365553 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.366798 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.367478 1994 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.367863 4895 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.368665 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.368841 4832 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.371312 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.368701 4895 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.371943 1994 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 3: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.372604 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.373301 1994 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.373528 4895 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.370949 4832 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.373895 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.375012 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.375195 1994 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.375790 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:12.370198 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3639394319998246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20250626 08:24:12.376608 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3619122549998792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2926s for a response
W20250626 08:24:12.377089 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3619122549998792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2926s for a response
I20250626 08:24:12.378226 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.379238 1998 raft_consensus.cc:2391] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.351961 4634 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.382208 4819 raft_consensus.cc:3053] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:12.358781 4581 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:12.370450 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:12.381697 2002 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.382642 4746 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.382495 4634 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.391855 4625 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.392452 4634 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.382669 4819 raft_consensus.cc:738] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:12.385546 1923 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 2.
I20250626 08:24:12.388017 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 637) took 1501 ms. Trace:
I20250626 08:24:12.387931 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.388602 4581 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:12.393764 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.393743 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.398747 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.398998 1999 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:12.389537 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.400420 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3081394320001891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20250626 08:24:12.400782 1893 connection.cc:232] RPC call timeout handler was delayed by -1.2685394320001251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20250626 08:24:12.401036 1893 connection.cc:232] RPC call timeout handler was delayed by -1.24063943200008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
I20250626 08:24:12.400223 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.401803 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.402809 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.401229 1893 connection.cc:232] RPC call timeout handler was delayed by -1.230739432000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
W20250626 08:24:12.403527 1893 connection.cc:232] RPC call timeout handler was delayed by -1.230739432000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
I20250626 08:24:12.388576 4629 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.405473 2002 raft_consensus.cc:2391] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.413442 1995 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
W20250626 08:24:12.414498 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1470394319999286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20250626 08:24:12.414515 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.396158 2586 rpcz_store.cc:276] 0626 08:24:10.854646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.855548 (+ 902us) service_pool.cc:225] Handling call
0626 08:24:10.856154 (+ 606us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:10.856810 (+ 656us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.856838 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:10.857442 (+ 604us) log.cc:844] Serialized 26 byte log entry
0626 08:24:11.979627 (+1122185us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:11.979840 (+ 213us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:11.980055 (+ 215us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.355461 (+375406us) raft_consensus.cc:1683] finished
0626 08:24:12.355480 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.355598 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1350,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":226,"thread_start_us":61262,"threads_started":1,"wal-append.queue_time_us":1247946}
I20250626 08:24:12.416723 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.418025 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.419253 1845 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 8: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.419348 1965 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.419454 1921 raft_consensus.cc:1238] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:12.423223 1841 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:12.424536 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.422621 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:12.426554 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 50855.2us
I20250626 08:24:12.428018 1999 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.429409 1995 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:12.430182 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.430464 1998 raft_consensus.cc:2408] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250626 08:24:12.431151 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.431615 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.434135 1928 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:12.419189 4671 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.436388 1994 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.437444 1967 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.442421 4746 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:12.442924 4746 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.443344 4746 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.443632 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.443681 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.435061 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.446055 4918 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 2 FOLLOWER]: Advancing to term 3
I20250626 08:24:12.446494 1844 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250626 08:24:12.447820 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.445987 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.450820 2002 raft_consensus.cc:1238] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250626 08:24:12.452148 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.452800 1921 raft_consensus.cc:2374] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4. Re-sending same reply.
I20250626 08:24:12.429008 4486 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:12.428953 4832 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.456382 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.456682 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2165) took 4808 ms. Trace:
I20250626 08:24:12.455561 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.456359 1994 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250626 08:24:12.457154 2002 rpcz_store.cc:276] 0626 08:24:07.645517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.418326 (+4772809us) service_pool.cc:225] Handling call
0626 08:24:12.434727 (+ 16401us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.453565 (+ 18838us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.453727 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.458367 1928 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:12.459012 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2171) took 3054 ms. Trace:
I20250626 08:24:12.459128 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 400) took 3011 ms. Trace:
I20250626 08:24:12.458916 4643 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:12.459548 1994 rpcz_store.cc:276] 0626 08:24:09.403119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.437316 (+3034197us) service_pool.cc:225] Handling call
0626 08:24:12.454420 (+ 17104us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.457549 (+ 3129us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.457711 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.460534 4581 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.435971 4840 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.451640 4625 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.011s
I20250626 08:24:12.462586 1995 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250626 08:24:12.463419 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2166) took 3066 ms. Trace:
I20250626 08:24:12.465431 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:12.466718 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 114842us
I20250626 08:24:12.465929 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.466325 2002 rpcz_store.cc:276] 0626 08:24:09.447607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.458186 (+3010579us) service_pool.cc:225] Handling call
0626 08:24:12.458552 (+ 366us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.458770 (+ 218us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.458794 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.458868 (+ 74us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:12.458902 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.458955 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.459061 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.471779 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.472256 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.472770 1921 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:12.473067 1923 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:12.473649 4916 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:12.474260 4916 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.474725 4916 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:12.475438 2550 leader_election.cc:336] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 2.035s (SENT)
I20250626 08:24:12.475522 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:12.476023 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.476558 1847 raft_consensus.cc:2391] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 8: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.477113 1998 raft_consensus.cc:1273] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:12.477507 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.477862 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2167) took 3080 ms. Trace:
I20250626 08:24:12.478375 1998 rpcz_store.cc:276] 0626 08:24:09.397192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.436480 (+3039288us) service_pool.cc:225] Handling call
0626 08:24:12.476671 (+ 40191us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.477570 (+ 899us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.477786 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.480515 1816 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.482323 4643 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:12.482815 4643 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.483193 4643 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:12.485023 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 114664us
I20250626 08:24:12.466300 1995 rpcz_store.cc:276] 0626 08:24:09.396895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.432158 (+3035263us) service_pool.cc:225] Handling call
0626 08:24:12.462173 (+ 30015us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.463150 (+ 977us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.463358 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.480641 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.490363 1845 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.464187 4819 consensus_queue.cc:260] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.495596 4819 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.498742 1967 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.502424 4719 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.489698 1995 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:12.506135 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 71115.4us
I20250626 08:24:12.534255 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 406) took 3051 ms. Trace:
I20250626 08:24:12.506693 4671 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:12.507215 4486 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.508502 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.509191 1841 raft_consensus.cc:2391] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.509423 4581 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.511448 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.511751 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.512431 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.513168 1925 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:12.513991 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.514336 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:12.515834 4920 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20250626 08:24:12.541065 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.538239 1924 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 2.
I20250626 08:24:12.513600 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 407) took 3030 ms. Trace:
I20250626 08:24:12.518934 2551 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.527813 4662 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:12.514012 4922 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:12.529363 1996 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.530071 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.530294 1999 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:12.530582 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:12.523404 1968 leader_election.cc:336] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.872s (SENT)
I20250626 08:24:12.519151 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.531234 1997 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:12.534608 1995 rpcz_store.cc:276] 0626 08:24:09.482611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.489035 (+3006424us) service_pool.cc:225] Handling call
0626 08:24:12.489425 (+ 390us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.534024 (+ 44599us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.534203 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.543129 1998 rpcz_store.cc:276] 0626 08:24:09.482852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.510163 (+3027311us) service_pool.cc:225] Handling call
0626 08:24:12.510488 (+ 325us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.510728 (+ 240us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.510753 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.512165 (+ 1412us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:12.512200 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.512251 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.513536 (+ 1285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.549891 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.552918 1845 raft_consensus.cc:2391] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 9: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.555131 1849 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:12.541946 1925 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:12.564347 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.565549 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.565883 4928 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:12.546618 1968 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.566560 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.535734 4671 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.568856 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.571393 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.572409 2551 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.573179 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2048) took 3031 ms. Trace:
I20250626 08:24:12.575718 1921 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
W20250626 08:24:12.573947 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 54942.5us
I20250626 08:24:12.579058 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.551478 4728 raft_consensus.cc:2802] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:12.580873 1995 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20250626 08:24:12.582446 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.583271 1923 raft_consensus.cc:2391] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:12.582630 1999 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250626 08:24:12.583910 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2178) took 2844 ms. Trace:
I20250626 08:24:12.584146 1999 rpcz_store.cc:276] 0626 08:24:09.739140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.555083 (+2815943us) service_pool.cc:225] Handling call
0626 08:24:12.578276 (+ 23193us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.583695 (+ 5419us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.583866 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.519621 1742 catalog_manager.cc:5582] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } }
I20250626 08:24:12.585249 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 410) took 2781 ms. Trace:
I20250626 08:24:12.585508 1995 rpcz_store.cc:276] 0626 08:24:09.804002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.555298 (+2751296us) service_pool.cc:225] Handling call
0626 08:24:12.574622 (+ 19324us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:12.581196 (+ 6574us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.585204 (+ 4008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.586423 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.586568 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.587316 1918 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:12.587528 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.587827 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.566550 4928 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:12.536916 4634 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20250626 08:24:12.594780 1928 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:12.595435 1925 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:12.571393 4671 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.541244 4629 consensus_queue.cc:1035] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20250626 08:24:12.587539 1920 raft_consensus.cc:2374] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6. Re-sending same reply.
I20250626 08:24:12.598440 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 411) took 2794 ms. Trace:
I20250626 08:24:12.598838 1998 rpcz_store.cc:276] 0626 08:24:09.804292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.563808 (+2759516us) service_pool.cc:225] Handling call
0626 08:24:12.592252 (+ 28444us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.592389 (+ 137us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.592404 (+ 15us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.598203 (+ 5799us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:12.598247 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.598305 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.598398 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.599224 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.600769 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.601645 2580 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:12.603547 4486 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.596829 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.606786 2586 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.607801 1965 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.608181 1893 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.606344 1997 raft_consensus.cc:1238] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250626 08:24:12.610651 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2179) took 2870 ms. Trace:
I20250626 08:24:12.610715 1924 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:12.611176 1997 rpcz_store.cc:276] 0626 08:24:09.740414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.569596 (+2829182us) service_pool.cc:225] Handling call
0626 08:24:12.578658 (+ 9062us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.610399 (+ 31741us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.610597 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.583127 4922 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:12.612104 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.616333 1996 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:12.616983 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 408) took 3133 ms. Trace:
I20250626 08:24:12.617331 1996 rpcz_store.cc:276] 0626 08:24:09.483072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.568154 (+3085082us) service_pool.cc:225] Handling call
0626 08:24:12.587272 (+ 19118us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.616758 (+ 29486us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.616926 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:12.618958 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.594990 4928 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:12.620066 1918 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:12.572836 4581 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.621217 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.623395 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.606858 1742 catalog_manager.cc:5582] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:24:12.630481 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.631381 4581 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.573596 1926 rpcz_store.cc:276] 0626 08:24:09.536512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.040624 (+2504112us) service_pool.cc:225] Handling call
0626 08:24:12.067764 (+ 27140us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.100980 (+ 33216us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:24:12.395827 (+294847us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.567302 (+171475us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.567367 (+ 65us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:12.567427 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.567496 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.567653 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15687,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":90,"raft.queue_time_us":30076,"thread_start_us":390300,"threads_started":2}
I20250626 08:24:12.631916 4581 consensus_queue.cc:1046] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 08:24:12.613744 4917 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:12.634686 1999 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
W20250626 08:24:12.627369 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 50218.3us
I20250626 08:24:12.635310 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 412) took 2830 ms. Trace:
I20250626 08:24:12.601655 4719 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.635684 1999 rpcz_store.cc:276] 0626 08:24:09.804554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.584665 (+2780111us) service_pool.cc:225] Handling call
0626 08:24:12.624988 (+ 40323us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.635065 (+ 10077us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.635259 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230144}
I20250626 08:24:12.626713 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2160) took 5106 ms. Trace:
I20250626 08:24:12.623502 4486 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:12.634080 4919 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:12.613827 4922 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.647909 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 413) took 2843 ms. Trace:
I20250626 08:24:12.640265 2001 rpcz_store.cc:276] 0626 08:24:07.515805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.170402 (+4654597us) service_pool.cc:225] Handling call
0626 08:24:12.200994 (+ 30592us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.202818 (+ 1824us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:24:12.466496 (+263678us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.615767 (+149271us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.615814 (+ 47us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:12.621601 (+ 5787us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.621651 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.621724 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2642,"prepare.queue_time_us":362402,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":31388,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":293996,"threads_started":2}
I20250626 08:24:12.648386 1995 rpcz_store.cc:276] 0626 08:24:09.804807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.592546 (+2787739us) service_pool.cc:225] Handling call
0626 08:24:12.624920 (+ 32374us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.642562 (+ 17642us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.642601 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.647635 (+ 5034us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:12.647681 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.647761 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.647860 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:12.651839 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 128459us
I20250626 08:24:12.633409 4917 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.655529 4917 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:12.655725 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.654523 1967 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.656754 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2159) took 5466 ms. Trace:
I20250626 08:24:12.657666 4495 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:12.658314 4495 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.658282 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.658943 4495 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.662989 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.663555 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.664105 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.664970 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.665201 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.666286 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.666486 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.657192 1993 rpcz_store.cc:276] 0626 08:24:07.190756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:10.846046 (+3655290us) service_pool.cc:225] Handling call
0626 08:24:10.846603 (+ 557us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:10.890017 (+ 43414us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:10.890065 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:12.041489 (+1151424us) log.cc:844] Serialized 26 byte log entry
0626 08:24:12.362341 (+320852us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:12.546572 (+184231us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.560304 (+ 13732us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.560432 (+ 128us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.656564 (+ 96132us) raft_consensus.cc:1683] finished
0626 08:24:12.656582 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.656701 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22664,"prepare.queue_time_us":238176,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":21037,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":5569,"thread_start_us":436977,"threads_started":2,"wal-append.queue_time_us":322598}
I20250626 08:24:12.667061 2584 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:12.662696 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.668669 4667 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:12.670929 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.668654 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.671694 2584 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:12.671777 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.658085 4895 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:12.635048 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.691111 2580 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 7: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:12.675942 1814 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:12.693472 1814 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:12.669420 4667 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.642352 4634 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:12.681097 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:12.699492 1849 raft_consensus.cc:604] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:12.644884 4919 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.682142 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1897) took 2198 ms. Trace:
I20250626 08:24:12.682884 1919 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 2. Current term is 3. Ops: []
I20250626 08:24:12.688872 4918 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250626 08:24:12.708751 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:12.677327 4895 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:12.703168 4667 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.714506 4895 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.715027 2578 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250626 08:24:12.650815 1742 catalog_manager.cc:5582] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 3 to 5, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 5 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 08:24:12.702023 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 49247.2us
I20250626 08:24:12.699916 1849 raft_consensus.cc:612] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:12.707376 4643 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:12.708504 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45904.4us
I20250626 08:24:12.702997 1927 rpcz_store.cc:276] 0626 08:24:10.483374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.091751 (+1608377us) service_pool.cc:225] Handling call
0626 08:24:12.092225 (+ 474us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.358786 (+266561us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.358816 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:12.389141 (+ 30325us) log.cc:844] Serialized 26 byte log entry
0626 08:24:12.538071 (+148930us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:12.538103 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.538165 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.681819 (+143654us) raft_consensus.cc:1683] finished
0626 08:24:12.681849 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.682063 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2710,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":298,"thread_start_us":372862,"threads_started":2,"wal-append.queue_time_us":188914}
I20250626 08:24:12.652881 4922 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:12.709717 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 85947.1us
I20250626 08:24:12.701066 4634 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.706189 4919 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.699615 4629 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:12.742803 4634 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:12.752020 4629 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.752503 4629 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.746055 4807 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:12.754925 4807 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.755321 4807 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.736266 4746 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.746177 1742 catalog_manager.cc:5582] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.762912 1742 catalog_manager.cc:5582] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 5 to 7, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 7 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.750695 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.765156 4671 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.766144 2586 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:12.768006 4495 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.705461 4728 raft_consensus.cc:695] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:12.778347 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 168817us
I20250626 08:24:12.786988 1925 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:12.787853 4832 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:12.789135 4671 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.790719 4662 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:12.798015 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 401) took 3349 ms. Trace:
I20250626 08:24:12.790545 4667 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.791885 4495 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.793380 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:12.794445 1742 catalog_manager.cc:5582] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 3 to 6, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 6 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.804080 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 405) took 3328 ms. Trace:
I20250626 08:24:12.803855 1742 catalog_manager.cc:5582] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.802176 4819 consensus_queue.cc:1046] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:24:12.804545 2002 rpcz_store.cc:276] 0626 08:24:09.475701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.530783 (+3055082us) service_pool.cc:225] Handling call
0626 08:24:12.562753 (+ 31970us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.633485 (+ 70732us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.633518 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:12.634029 (+ 511us) log.cc:844] Serialized 26 byte log entry
0626 08:24:12.705252 (+ 71223us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:12.705292 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.705364 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.803870 (+ 98506us) raft_consensus.cc:1683] finished
0626 08:24:12.803890 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.804025 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19695,"raft.queue_time_us":20724,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":70811,"threads_started":1,"wal-append.queue_time_us":167938}
I20250626 08:24:12.806363 4819 raft_consensus.cc:3053] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:12.800137 4911 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:12.800551 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:12.811023 1849 raft_consensus.cc:604] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:12.790060 4916 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.800659 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:12.812522 4932 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250626 08:24:12.813417 4916 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:12.814128 4495 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:12.814409 4625 consensus_queue.cc:1035] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:24:12.815747 1967 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.815927 4625 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 08:24:12.816507 4495 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:12.816916 4495 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:12.817262 4495 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.817315 4625 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250626 08:24:12.798418 1994 rpcz_store.cc:276] 0626 08:24:09.448896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.460500 (+3011604us) service_pool.cc:225] Handling call
0626 08:24:12.460936 (+ 436us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:12.461221 (+ 285us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:24:12.538577 (+ 77356us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.797418 (+258841us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:12.797472 (+ 54us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:12.797595 (+ 123us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.797818 (+ 223us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.797954 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":139498,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":2795,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":282163,"threads_started":2}
I20250626 08:24:12.801831 4832 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:12.822772 4832 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:12.796999 1923 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:12.799613 4643 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:12.828970 1890 connection.cc:232] RPC call timeout handler was delayed by -1.8430585390001397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20250626 08:24:12.829375 1890 connection.cc:232] RPC call timeout handler was delayed by -1.7197585390003951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
I20250626 08:24:12.829432 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2162) took 5185 ms. Trace:
I20250626 08:24:12.805765 1742 catalog_manager.cc:5582] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 2, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 2 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.832087 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.832690 2586 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:12.833580 1965 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.825491 4819 raft_consensus.cc:738] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:12.829869 2000 rpcz_store.cc:276] 0626 08:24:07.644352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:11.008046 (+3363694us) service_pool.cc:225] Handling call
0626 08:24:11.996111 (+988065us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.031373 (+ 35262us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.031420 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:12.032046 (+ 626us) log.cc:844] Serialized 26 byte log entry
0626 08:24:12.301112 (+269066us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:12.571675 (+270563us) log.cc:844] Serialized 14 byte log entry
0626 08:24:12.603974 (+ 32299us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b180002eae0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:12.829111 (+225137us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:12.829176 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:12.829221 (+ 45us) raft_consensus.cc:1683] finished
0626 08:24:12.829314 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:12.829389 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28899,"prepare.queue_time_us":274341,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":3334,"raft.queue_time_us":23155,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":3278,"spinlock_wait_cycles":15122816,"thread_start_us":791163,"threads_started":3,"wal-append.queue_time_us":268999}
I20250626 08:24:12.772554 4634 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.835727 1742 catalog_manager.cc:5582] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 3 to 5, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 5 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.843073 1849 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8e262c3543404a9cb840169098735ea2"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:12.843717 1849 raft_consensus.cc:604] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:12.844054 1849 raft_consensus.cc:612] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:12.858876 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.859992 4719 raft_consensus.cc:3053] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:12.860383 4719 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:12.860759 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.861374 1999 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:12.861243 4719 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.862323 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.862498 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.862923 1999 raft_consensus.cc:2374] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6. Re-sending same reply.
I20250626 08:24:12.863657 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.863893 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.863947 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.862651 4719 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.864599 1997 raft_consensus.cc:2374] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7. Re-sending same reply.
I20250626 08:24:12.865957 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.867786 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 243356us
I20250626 08:24:12.868669 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.869433 1919 raft_consensus.cc:3058] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 7
I20250626 08:24:12.868669 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.850975 4667 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.870113 1849 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:12.870209 1995 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:12.866004 4643 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20250626 08:24:12.871253 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.871686 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.872239 1993 raft_consensus.cc:2374] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9. Re-sending same reply.
I20250626 08:24:12.873220 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.873476 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.874097 1996 raft_consensus.cc:2391] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.873476 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.875175 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.872342 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.875723 2001 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:12.877229 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.839809 4916 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:12.867506 2002 raft_consensus.cc:2374] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4. Re-sending same reply.
W20250626 08:24:12.882766 2550 leader_election.cc:336] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813): Timed out: RequestConsensusVote RPC to 127.0.171.194:39813 timed out after 1.866s (SENT)
I20250626 08:24:12.882915 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:12.885164 1814 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:12.878365 4938 consensus_queue.cc:1046] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250626 08:24:12.887187 4939 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
W20250626 08:24:12.887635 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.888418 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.888972 4916 raft_consensus.cc:695] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:12.889325 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.890144 1843 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:12.890228 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.890920 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:12.891319 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.891654 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:12.890987 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.892199 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.892517 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.892894 1842 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:12.893242 1848 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:12.888996 1847 raft_consensus.cc:2374] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9. Re-sending same reply.
I20250626 08:24:12.892225 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.895008 2002 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.895602 4629 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.896131 1998 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:24:12.897240 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.897567 4629 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.898252 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.898376 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.899462 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.900030 2002 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:12.900331 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.901039 1993 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:12.900970 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:12.901525 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.901901 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.902045 1890 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.890988 1845 raft_consensus.cc:2391] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.902500 1993 raft_consensus.cc:2374] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7. Re-sending same reply.
I20250626 08:24:12.897786 1998 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:12.903221 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.892225 1850 raft_consensus.cc:2374] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6. Re-sending same reply.
I20250626 08:24:12.903927 1999 raft_consensus.cc:2391] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:12.890009 4916 consensus_queue.cc:237] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.899489 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.905475 2001 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250626 08:24:12.905781 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.905716 1742 catalog_manager.cc:5582] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:12.908412 1994 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250626 08:24:12.910517 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.910535 2586 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:12.884733 4746 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:12.913254 1925 raft_consensus.cc:2391] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:12.914448 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:12.915331 4746 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:12.915807 4746 raft_consensus.cc:695] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:12.916596 4746 consensus_queue.cc:237] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.918185 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.918797 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.919739 4932 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:12.918803 1997 raft_consensus.cc:2374] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7. Re-sending same reply.
I20250626 08:24:12.917870 4728 consensus_queue.cc:237] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.925069 1919 raft_consensus.cc:2466] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:12.933367 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:12.934155 2002 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:12.935508 4922 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:12.937543 4922 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:12.938243 2587 raft_consensus.cc:2366] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:12.938297 2000 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250626 08:24:12.939617 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.965677 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:12.971566 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.971796 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.972898 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:12.974162 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:12.975538 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.976217 1997 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:12.975018 1919 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 2 FOLLOWER]: Advancing to term 4
W20250626 08:24:12.978116 1968 leader_election.cc:336] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.260s (SENT)
I20250626 08:24:12.978750 1968 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:12.978561 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 92014.5us
I20250626 08:24:12.979939 1967 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:12.910517 4941 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:12.981633 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48435us
I20250626 08:24:12.933547 4643 raft_consensus.cc:3053] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:12.988927 1850 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250626 08:24:12.926817 4932 raft_consensus.cc:695] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:12.995640 1742 catalog_manager.cc:5582] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 3, leader changed from be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:24:12.999603 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 166907us
I20250626 08:24:13.000003 4948 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:13.000017 1742 catalog_manager.cc:5582] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:13.003026 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.002868 1742 catalog_manager.cc:5582] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:13.004488 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.004645 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.005113 1845 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:13.006351 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.007077 1845 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
I20250626 08:24:13.007326 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:13.011304 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.000470 1893 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:13.015277 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 150174us
I20250626 08:24:12.917791 4919 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.021868 1919 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:12.985368 4941 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:13.004949 1742 catalog_manager.cc:5582] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 1 to 3, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 3 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:13.027247 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:12.911139 4819 consensus_queue.cc:260] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:12.989979 4643 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:13.035377 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:12.939273 4928 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.037243 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.038230 1849 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 8: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:13.038769 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 60687.5us
I20250626 08:24:13.000726 4911 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:13.040416 1742 catalog_manager.cc:5582] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:13.038961 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:13.042408 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 43955.2us
I20250626 08:24:13.012610 4948 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:13.016439 4662 raft_consensus.cc:2802] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:13.045163 4945 consensus_queue.cc:1035] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.005s
I20250626 08:24:13.045326 1742 catalog_manager.cc:5582] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196). New cstate: current_term: 4 leader_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:13.046844 1847 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:13.050963 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.057968 4942 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250626 08:24:13.071828 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.072741 4928 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:13.073336 4643 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:13.074743 4643 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.077252 2000 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
W20250626 08:24:13.408452 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:13.079648 1742 catalog_manager.cc:5582] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } } }
W20250626 08:24:13.408106 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 364520us
I20250626 08:24:13.036054 4941 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.077819 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.412496 2002 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:13.114033 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 52540.2us
W20250626 08:24:13.457290 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 473739us
W20250626 08:24:13.455147 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 366904us
W20250626 08:24:13.395412 4495 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.004s
I20250626 08:24:13.054890 4948 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:13.463289 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:13.456013 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:13.375283 4807 consensus_meta.cc:287] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.619s user 0.002s sys 0.009s
W20250626 08:24:13.375556 4895 consensus_meta.cc:287] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.661s user 0.010s sys 0.000s
W20250626 08:24:13.475016 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 63426.6us
W20250626 08:24:13.464916 4728 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.002s sys 0.028s
I20250626 08:24:13.472654 4895 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.484056 4918 raft_consensus.cc:3053] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:13.179127 4629 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.301s
I20250626 08:24:13.486954 2002 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
W20250626 08:24:13.486277 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 447365us
W20250626 08:24:13.482350 4728 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.002s sys 0.029s
I20250626 08:24:13.113576 4832 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.474964 4671 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:13.504199 4634 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:13.467336 4495 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:13.505864 4840 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:13.239591 4932 consensus_queue.cc:237] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.508181 1996 raft_consensus.cc:2366] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
I20250626 08:24:13.516834 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:13.517506 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:13.518239 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.519158 1994 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:13.519217 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.519840 2002 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.520495 1995 raft_consensus.cc:1273] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:13.518312 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:13.528383 2002 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:24:13.540127 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 53610.7us
I20250626 08:24:13.491112 4918 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:13.493888 4919 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:13.548386 1816 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:13.507889 4495 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:13.551019 4955 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:13.551371 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:13.551584 4955 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:13.552012 2587 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:13.470336 4807 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:13.561158 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 64982.1us
W20250626 08:24:13.566825 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 110526us
I20250626 08:24:13.562299 4895 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:13.583941 1841 raft_consensus.cc:2366] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
W20250626 08:24:13.561467 4917 consensus_meta.cc:287] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.905s user 0.005s sys 0.006s
I20250626 08:24:13.580467 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.594084 1890 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:13.552012 4955 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.543336 4662 raft_consensus.cc:695] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:13.604580 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 134059us
W20250626 08:24:13.616344 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:13.664271 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 52519.6us
I20250626 08:24:13.585222 4643 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:13.713052 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 82701.7us
W20250626 08:24:13.662005 4936 log.cc:927] Time spent T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.500s user 0.001s sys 0.000s
W20250626 08:24:13.719553 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 93039.3us
I20250626 08:24:13.720507 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.721521 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:13.722337 4946 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250626 08:24:13.724913 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 183729us
I20250626 08:24:13.632892 4956 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:13.725617 1890 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:13.727676 2581 raft_consensus.cc:2366] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
W20250626 08:24:13.728255 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.002s sys 0.008s
W20250626 08:24:13.728626 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.002s sys 0.008s
W20250626 08:24:13.729372 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 728920us
I20250626 08:24:13.611894 4917 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.732192 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.739051 4919 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:13.742692 4956 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:13.746438 4807 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:13.721521 4819 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:13.626207 4942 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20250626 08:24:13.749886 2586 raft_consensus.cc:2366] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7.
W20250626 08:24:13.766966 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 271803us
I20250626 08:24:13.786180 1967 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:13.792682 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:13.727476 4946 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:13.799183 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.754650 4919 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:13.806546 4946 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:13.853394 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.718703 4918 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:13.855558 4918 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:13.855624 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 101170us
I20250626 08:24:13.858247 1816 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:13.859153 4807 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250626 08:24:13.859926 1816 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:13.861476 4973 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:13.862135 4973 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:13.864679 4973 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:13.865545 4807 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:13.866070 4807 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:13.866490 4807 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.868002 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:13.870680 4643 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250626 08:24:13.875723 4949 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.321s user 0.003s sys 0.000s
W20250626 08:24:13.879201 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:13.879001 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 116150us
I20250626 08:24:13.884429 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:13.841926 4919 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:13.890669 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.891554 2002 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:24:13.894809 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 108741us
I20250626 08:24:13.896492 1816 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:13.904268 2587 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:13.905468 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:13.906419 1965 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:13.904467 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:13.907902 1923 raft_consensus.cc:2408] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:13.910547 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.911314 2000 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:13.891328 4486 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:13.923314 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:13.901087 4911 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:13.927217 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:13.935050 1919 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:13.889700 4922 consensus_queue.cc:1035] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20250626 08:24:13.895843 4728 consensus_queue.cc:1046] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20250626 08:24:13.949577 4832 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:13.950469 4832 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:13.950843 4832 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:13.951256 4832 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:13.954499 4917 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:13.954229 4978 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20250626 08:24:13.955395 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:13.913843 4643 consensus_queue.cc:1046] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250626 08:24:13.970392 4916 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:13.971056 4916 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:13.974321 1890 leader_election.cc:400] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
I20250626 08:24:13.974685 1890 leader_election.cc:403] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20250626 08:24:13.975248 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 69681.9us
W20250626 08:24:13.976480 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:13.938540 4486 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:13.987437 1919 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:13.995103 1967 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:13.943688 4911 raft_consensus.cc:695] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:13.901274 4662 consensus_queue.cc:237] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.042227 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.044412 2587 raft_consensus.cc:1238] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:14.044050 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2190) took 1147 ms. Trace:
I20250626 08:24:14.067303 1996 rpcz_store.cc:276] 0626 08:24:12.896703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.899169 (+ 2466us) service_pool.cc:225] Handling call
0626 08:24:14.043847 (+1144678us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b5400203ad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:14.043997 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1398219776}
W20250626 08:24:14.017133 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 83129us
I20250626 08:24:14.070181 2550 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:13.970441 4966 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:14.073835 2550 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:14.004618 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:14.075250 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:14.044915 4719 consensus_meta.cc:287] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.166s user 0.006s sys 0.004s
W20250626 08:24:14.084607 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 76795us
W20250626 08:24:14.018568 4975 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20250626 08:24:14.090013 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 233357us
W20250626 08:24:14.016939 4957 log.cc:927] Time spent T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.233s user 0.000s sys 0.001s
I20250626 08:24:14.089501 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.092020 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:14.079322 4922 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:14.093405 4961 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:14.043007 4916 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:14.079684 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:14.017431 4940 raft_consensus.cc:2802] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:14.075244 4840 raft_consensus.cc:3053] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:14.103060 1849 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:24:14.099263 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 102150us
W20250626 08:24:14.103763 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 101009us
I20250626 08:24:14.113090 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.114001 2582 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEARNER]: Advancing to term 5
I20250626 08:24:14.110594 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.114691 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:14.111431 4948 consensus_meta.cc:287] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.638s user 0.012s sys 0.000s
I20250626 08:24:14.118615 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.119539 1919 raft_consensus.cc:2408] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250626 08:24:14.120564 1925 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9.
I20250626 08:24:14.022169 4987 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 3. Current term is 4.
W20250626 08:24:14.127947 1968 leader_election.cc:336] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.713s (SENT)
I20250626 08:24:14.126267 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.023460 4955 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.029656 4946 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.138962 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.122179 1967 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.113171 4966 raft_consensus.cc:695] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:14.144642 4966 consensus_queue.cc:237] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.108613 4956 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:14.112743 4840 raft_consensus.cc:738] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:14.131831 4941 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.666s user 0.009s sys 0.002s
I20250626 08:24:14.172838 1927 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20250626 08:24:14.155694 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 134717us
I20250626 08:24:14.184377 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2199) took 1156 ms. Trace:
W20250626 08:24:14.182687 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 64030.4us
I20250626 08:24:14.184823 1998 rpcz_store.cc:276] 0626 08:24:13.027627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:13.037324 (+ 9697us) service_pool.cc:225] Handling call
0626 08:24:13.046290 (+ 8966us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:13.487763 (+441473us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:13.487813 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:13.510642 (+ 22829us) log.cc:844] Serialized 26 byte log entry
0626 08:24:13.708157 (+197515us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:13.925058 (+216901us) log.cc:844] Serialized 14 byte log entry
0626 08:24:13.938783 (+ 13725us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:13.938863 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:14.184141 (+245278us) raft_consensus.cc:1683] finished
0626 08:24:14.184165 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:14.184315 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":128071,"prepare.queue_time_us":326508,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":497389,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":51086,"thread_start_us":647165,"threads_started":3,"wal-append.queue_time_us":312689}
I20250626 08:24:14.138010 4961 raft_consensus.cc:695] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:14.191946 1968 leader_election.cc:336] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.529s (SENT)
I20250626 08:24:14.151854 4987 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:14.149518 4940 raft_consensus.cc:695] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:14.127010 4948 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.195549 2000 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:14.177556 4916 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250626 08:24:14.163511 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:14.177006 4917 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:14.212970 1998 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:14.214497 1925 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:14.171029 4807 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.178437 4643 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:14.207513 1968 leader_election.cc:336] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.563s (SENT)
I20250626 08:24:14.210058 4992 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:14.126650 4486 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.221539 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.201550 1816 leader_election.cc:400] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
I20250626 08:24:14.214574 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:14.222653 1816 leader_election.cc:403] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:14.223022 1923 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:14.220412 1926 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:14.206658 2579 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:14.205706 4916 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:14.225963 4992 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.226953 4979 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:14.150889 4911 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.230129 4954 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:14.231419 4948 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:14.223713 4643 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.232973 2578 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:14.310015 4954 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.310982 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.312245 4954 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:14.233821 4940 consensus_queue.cc:237] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:14.234939 4968 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250626 08:24:14.215227 4980 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:14.283651 4865 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:14.237663 2582 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEARNER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:24:14.258821 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 57358.9us
I20250626 08:24:14.260506 4974 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:14.261576 4981 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.231s user 0.000s sys 0.003s
I20250626 08:24:14.183840 4941 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:14.286137 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 76847.5us
I20250626 08:24:14.216521 4917 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:14.265247 4975 log.cc:927] Time spent T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250626 08:24:14.222059 4989 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:14.199968 4987 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:14.318217 1968 leader_election.cc:336] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.626s (SENT)
I20250626 08:24:14.334147 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:14.342682 1968 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.284474 4916 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:14.322070 4980 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:14.324450 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:14.352289 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 233014us
I20250626 08:24:14.353679 2582 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:14.325537 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:14.333017 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 197600us
I20250626 08:24:14.335343 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.355412 1923 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:14.356464 1923 raft_consensus.cc:1238] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:14.343389 1926 raft_consensus.cc:2391] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:14.319018 4643 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:14.237442 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:14.358381 2587 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:14.358966 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:14.359416 1928 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:14.359714 1918 raft_consensus.cc:2391] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:14.351466 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.360368 1925 raft_consensus.cc:2391] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:14.328416 4917 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:14.385026 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.320169 4956 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20250626 08:24:14.392021 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:14.392051 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.398851 2587 raft_consensus.cc:1273] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:24:14.402683 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.390899 2582 pending_rounds.cc:85] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Aborting all ops after (but not including) 5
I20250626 08:24:14.362782 4946 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:14.329469 4963 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.066s
W20250626 08:24:14.406960 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 88787.1us
I20250626 08:24:14.404386 2582 pending_rounds.cc:107] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250626 08:24:14.359206 4980 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:14.406968 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2201) took 1333 ms. Trace:
I20250626 08:24:14.402194 1893 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.388279 4955 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:14.410781 1968 leader_election.cc:336] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.647s (SENT)
I20250626 08:24:14.410706 2582 raft_consensus.cc:2887] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:14.412403 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:14.411989 1999 rpcz_store.cc:276] 0626 08:24:13.073999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:13.076390 (+ 2391us) service_pool.cc:225] Handling call
0626 08:24:13.411355 (+334965us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:13.462171 (+ 50816us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:13.462214 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:13.469976 (+ 7762us) log.cc:844] Serialized 26 byte log entry
0626 08:24:13.586865 (+116889us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:13.586909 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:13.586986 (+ 77us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:14.406760 (+819774us) raft_consensus.cc:1683] finished
0626 08:24:14.406783 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:14.406912 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64403,"raft.queue_time_us":97347,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":10121,"thread_start_us":102843,"threads_started":1,"wal-append.queue_time_us":608214}
I20250626 08:24:14.409127 4895 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:14.425184 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 68708.4us
I20250626 08:24:14.426348 2551 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:14.350451 4989 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:14.436864 1968 leader_election.cc:336] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.702s (SENT)
I20250626 08:24:14.459712 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.460155 4941 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:14.460330 4955 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:14.460768 4496 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:14.464408 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.405084 4746 consensus_queue.cc:1035] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20250626 08:24:14.422371 4895 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:14.329741 4961 consensus_queue.cc:237] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.470324 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.472344 4992 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:14.472472 4959 raft_consensus.cc:2747] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:14.489971 1845 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:14.426843 4911 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:14.518551 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 81716.8us
I20250626 08:24:14.519063 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.520030 1848 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:14.522676 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:14.527842 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 101601us
I20250626 08:24:14.534286 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:14.535100 1999 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:14.534730 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 214256us
I20250626 08:24:14.541373 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.542519 1842 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:14.469521 4895 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:14.550148 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:14.465783 4807 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:14.518348 4991 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.293s user 0.001s sys 0.000s
W20250626 08:24:14.534515 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 202089us
I20250626 08:24:14.556753 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.562860 1848 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
W20250626 08:24:14.560191 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 97944.2us
I20250626 08:24:14.565853 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:14.538528 4949 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250626 08:24:14.571049 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.571240 1893 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:14.572185 1842 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:14.536445 4955 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:14.579232 1848 raft_consensus.cc:1238] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:14.580132 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.556578 4960 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:14.570801 4946 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:14.616672 1814 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:14.612040 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 82910.7us
W20250626 08:24:14.619449 1968 leader_election.cc:336] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.779s (SENT)
I20250626 08:24:14.620049 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.621016 4911 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:14.621466 1968 leader_election.cc:336] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.794s (SENT)
I20250626 08:24:14.622061 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.623034 4911 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250626 08:24:14.623728 1968 leader_election.cc:336] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.934s (SENT)
I20250626 08:24:14.624267 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.628939 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.633082 1999 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:14.645056 1846 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:14.645282 5005 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:14.645536 1845 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:14.645732 1846 raft_consensus.cc:604] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:14.645937 5005 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:14.646584 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.646956 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.647012 5005 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.647995 1846 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:14.647212 1845 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:14.647888 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.625376 4911 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:14.704865 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.701036 1844 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:24:14.701637 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 81670.8us
I20250626 08:24:14.688186 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.706648 1928 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:14.708274 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:14.709594 1919 raft_consensus.cc:1273] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250626 08:24:14.710078 2551 leader_election.cc:336] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.510s (SENT)
I20250626 08:24:14.617588 4946 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:14.711494 1816 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:14.713956 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 99823.3us
I20250626 08:24:14.714990 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.711966 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.715989 1846 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:14.716985 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.718669 1844 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:14.720136 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:14.713016 2551 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.719624 1893 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.720818 1925 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 9.
I20250626 08:24:14.618065 4807 consensus_queue.cc:1046] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20250626 08:24:14.579046 5000 consensus_queue.cc:1046] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.238s
I20250626 08:24:14.724983 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.723572 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.727572 1844 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:14.734326 2582 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:14.735549 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:14.738771 2551 leader_election.cc:336] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.572s (SENT)
I20250626 08:24:14.739696 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.717415 4946 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:14.747834 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.752636 1999 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:14.751540 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.752550 1893 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:14.754235 1845 raft_consensus.cc:2391] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:14.704367 4992 consensus_queue.cc:1046] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.028s
I20250626 08:24:14.613364 4955 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.753337 1846 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:14.768960 4989 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:14.618413 4960 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.721855 4987 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:14.774933 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.782428 1845 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:14.771485 4919 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.859s user 0.012s sys 0.000s
I20250626 08:24:14.796860 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.797529 1846 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:14.770668 2551 leader_election.cc:336] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.584s (SENT)
W20250626 08:24:14.780632 1968 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.696s (SENT) [suppressed 80 similar messages]
W20250626 08:24:14.781183 4918 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.891s user 0.000s sys 0.017s
W20250626 08:24:14.812951 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 44514.7us
W20250626 08:24:14.811913 1968 leader_election.cc:336] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.757s (SENT)
I20250626 08:24:14.816767 1968 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.774798 5007 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:14.820549 1848 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250626 08:24:14.821666 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 69186.2us
I20250626 08:24:14.823480 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.780418 5000 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:14.824558 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.826617 1845 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 7
I20250626 08:24:14.829612 1844 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:14.830582 1893 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:14.824455 2585 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:14.837949 5013 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:14.838233 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.838866 1846 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:14.839882 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:14.840394 5009 log.cc:927] Time spent T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
I20250626 08:24:14.840898 1846 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:14.841862 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2197) took 1863 ms. Trace:
I20250626 08:24:14.842437 1997 rpcz_store.cc:276] 0626 08:24:12.978056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.987461 (+ 9405us) service_pool.cc:225] Handling call
0626 08:24:13.000915 (+ 13454us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:13.064499 (+ 63584us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:13.064530 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:13.884926 (+820396us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b1800009280. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:14.263203 (+378277us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b2c00151928. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb7a226d1 00007f4eb7a31d02 00007f4eb90bac51 00007f4eb907f02a 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832
0626 08:24:14.263742 (+ 539us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.403648 (+139906us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:14.415380 (+ 11732us) log.cc:844] Serialized 14 byte log entry
0626 08:24:14.424880 (+ 9500us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.424958 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:14.841640 (+416682us) raft_consensus.cc:1683] finished
0626 08:24:14.841669 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:14.841803 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20210,"prepare.queue_time_us":12995,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1317,"raft.queue_time_us":129312,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":2018,"spinlock_wait_cycles":66173696,"thread_start_us":184519,"threads_started":2,"wal-append.queue_time_us":271366}
I20250626 08:24:14.841817 5019 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:14.839843 4999 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:14.844439 5019 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:14.844892 5019 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:14.843569 4963 consensus_queue.cc:1046] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:24:14.841692 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.852660 1846 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:14.838536 5013 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.856642 1893 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:14.844254 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:14.839818 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:14.848416 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.811177 4989 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
W20250626 08:24:14.855268 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 84629.5us
I20250626 08:24:14.813455 4807 raft_consensus.cc:3053] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
W20250626 08:24:14.834187 1968 leader_election.cc:336] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.995s (SENT)
I20250626 08:24:14.871644 1968 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.870196 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.864583 1844 raft_consensus.cc:2374] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6. Re-sending same reply.
I20250626 08:24:14.875599 1846 raft_consensus.cc:2374] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5. Re-sending same reply.
I20250626 08:24:14.889415 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.890748 1844 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9.
I20250626 08:24:14.831872 4496 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:14.892410 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:14.895329 4943 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.006s sys 0.023s
W20250626 08:24:14.895746 4943 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.153s user 0.006s sys 0.024s
W20250626 08:24:14.896891 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 62729.9us
I20250626 08:24:14.897743 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:14.897665 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 347623us
I20250626 08:24:14.898912 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.813109 4919 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:14.901752 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.853505 5007 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:14.902150 1968 leader_election.cc:336] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.195s (SENT)
I20250626 08:24:14.872565 4987 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 08:24:14.873991 4807 raft_consensus.cc:738] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:14.926357 1968 leader_election.cc:336] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.249s (SENT)
I20250626 08:24:14.927304 1968 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:14.892411 4911 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:14.910494 4832 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.959s user 0.013s sys 0.000s
I20250626 08:24:14.949028 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.950574 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.896122 5000 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:14.926249 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:14.937899 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 64174.6us
I20250626 08:24:14.901103 4989 raft_consensus.cc:695] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:14.947746 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.956223 1846 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:14.825428 5016 consensus_queue.cc:1046] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.295s
I20250626 08:24:14.949060 2550 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:14.959616 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 135628us
I20250626 08:24:14.960822 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.961464 1844 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:14.963310 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:14.966560 1997 raft_consensus.cc:1238] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:14.966806 2550 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:14.968479 1995 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:14.983798 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:14.946269 4496 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:14.931247 4941 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20250626 08:24:14.958065 4832 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:14.931391 4974 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20250626 08:24:14.981186 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:15.008232 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:15.014287 1812 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:15.025429 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 55393us
I20250626 08:24:15.007637 1848 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:15.008693 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:14.989754 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:15.070396 1999 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:14.931352 5004 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:14.987950 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:15.073936 1997 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:24:15.038517 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 112343us
W20250626 08:24:14.977838 5001 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.335s user 0.001s sys 0.001s
W20250626 08:24:15.015385 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 48670.9us
I20250626 08:24:15.008687 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:15.102833 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:15.055975 4865 consensus_meta.cc:287] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.733s user 0.011s sys 0.000s
I20250626 08:24:15.118237 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:15.119535 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:15.034437 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:15.120431 1920 raft_consensus.cc:1238] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:15.124933 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:15.036586 5006 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:24:15.086789 4971 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.004s sys 0.016s
I20250626 08:24:15.134539 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:15.040274 5025 raft_consensus.cc:3053] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:15.146981 1997 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:14.987227 4807 consensus_queue.cc:260] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:15.031980 5007 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:15.085214 4989 consensus_queue.cc:237] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:15.473345 2002 raft_consensus.cc:1273] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:24:15.226994 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 151907us
I20250626 08:24:15.293998 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 689) took 1208 ms. Trace:
I20250626 08:24:15.344559 1999 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
W20250626 08:24:15.568967 1841 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.004s sys 0.008s
I20250626 08:24:15.493959 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2075) took 1738 ms. Trace:
I20250626 08:24:15.237530 1925 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63"
dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:48988
I20250626 08:24:16.006665 1925 raft_consensus.cc:604] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:16.006318 1927 rpcz_store.cc:276] 0626 08:24:13.755691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:13.771244 (+ 15553us) service_pool.cc:225] Handling call
0626 08:24:13.771582 (+ 338us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.173422 (+401840us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:15.493579 (+1320157us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54002af3d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:15.493859 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":919117056}
I20250626 08:24:16.007119 1925 raft_consensus.cc:612] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250626 08:24:15.282286 1968 leader_election.cc:336] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.206s (SENT)
I20250626 08:24:16.007755 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:15.742638 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.008s sys 0.008s
I20250626 08:24:15.998418 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 579) took 1645 ms. Trace:
W20250626 08:24:15.898288 1849 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1791) took 2455 ms (client timeout 2437 ms). Trace:
W20250626 08:24:15.800374 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.005s sys 0.013s
W20250626 08:24:15.799795 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.018s sys 0.024s
W20250626 08:24:15.720472 1893 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.012s sys 0.004s
W20250626 08:24:16.010879 1893 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.035s user 0.016s sys 0.009s
W20250626 08:24:16.005657 1841 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.101s user 0.006s sys 0.018s
W20250626 08:24:16.011462 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.03815e+06us
W20250626 08:24:16.008389 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 726106us
W20250626 08:24:16.010439 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.011s sys 0.013s
I20250626 08:24:15.475278 1814 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:15.132826 4865 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
W20250626 08:24:15.790349 4976 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 1.797s user 0.012s sys 0.000s
W20250626 08:24:16.009037 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.860s user 0.010s sys 0.014s
W20250626 08:24:15.752066 1845 consensus_meta.cc:287] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.906s user 0.007s sys 0.004s
W20250626 08:24:16.023219 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 919259us
I20250626 08:24:15.290899 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:16.016142 1849 rpcz_store.cc:269] 0626 08:24:13.454944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.041791 (+586847us) service_pool.cc:225] Handling call
0626 08:24:15.898077 (+1856286us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b54001f1154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:15.898203 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84587776}
W20250626 08:24:15.892154 2585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 719) took 2306 ms (client timeout 2037 ms). Trace:
W20250626 08:24:16.010713 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.853s user 0.019s sys 0.025s
I20250626 08:24:15.209129 1848 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
W20250626 08:24:16.015789 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 540561us
I20250626 08:24:15.896826 1998 raft_consensus.cc:1238] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:15.475888 2586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 692) took 1356 ms. Trace:
W20250626 08:24:15.807037 4895 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.233s user 0.010s sys 0.003s
I20250626 08:24:15.104180 4960 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:15.337639 5012 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20250626 08:24:15.314378 5030 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:15.284101 4949 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
W20250626 08:24:15.337355 4993 log.cc:927] Time spent T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250626 08:24:16.075712 1997 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:16.078397 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.079123 1995 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
W20250626 08:24:15.583982 5026 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.531s user 0.000s sys 0.001s
W20250626 08:24:15.791180 4962 log.cc:927] Time spent T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 1.776s user 0.001s sys 0.006s
W20250626 08:24:15.471562 5008 log.cc:927] Time spent T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.533s user 0.000s sys 0.001s
I20250626 08:24:16.083138 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 523) took 1505 ms. Trace:
I20250626 08:24:16.083835 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 434) took 1976 ms. Trace:
I20250626 08:24:15.157348 5025 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:15.791553 4980 consensus_meta.cc:287] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.368s user 0.013s sys 0.000s
W20250626 08:24:15.598937 4946 consensus_meta.cc:287] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.819s user 0.011s sys 0.000s
W20250626 08:24:15.792333 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:15.584291 5010 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.470s user 0.000s sys 0.001s
W20250626 08:24:15.549748 4987 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.001s sys 0.011s
W20250626 08:24:15.895228 5019 consensus_meta.cc:287] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.050s user 0.009s sys 0.002s
W20250626 08:24:15.593719 4916 consensus_meta.cc:287] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.231s user 0.010s sys 0.000s
W20250626 08:24:15.695082 4917 consensus_meta.cc:287] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 1.287s user 0.010s sys 0.002s
W20250626 08:24:15.588449 5039 log.cc:927] Time spent T 00000000000000000000000000000000 P 12239b0901474fcc8d88fc20bc33dc1f: Append to log took a long time: real 0.412s user 0.002s sys 0.000s
W20250626 08:24:15.249706 5003 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.304s user 0.000s sys 0.001s
I20250626 08:24:16.061030 1847 raft_consensus.cc:2366] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:15.878018 4979 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:15.474987 2581 rpcz_store.cc:276] 0626 08:24:14.085754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.085940 (+ 186us) service_pool.cc:225] Handling call
0626 08:24:14.086327 (+ 387us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:14.107764 (+ 21437us) raft_consensus.cc:1532] Early marking committed up to index 8
0626 08:24:14.205808 (+ 98044us) log.cc:844] Serialized 14 byte log entry
0626 08:24:14.933075 (+727267us) log.cc:844] Serialized 14 byte log entry
0626 08:24:15.275850 (+342775us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:15.275913 (+ 63us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:15.293679 (+ 17766us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:15.293755 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:15.293933 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21385,"prepare.queue_time_us":218563,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":363359,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":13044,"thread_start_us":1104942,"threads_started":4,"wal-append.queue_time_us":462255}
I20250626 08:24:15.343705 5036 consensus_queue.cc:1046] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 1.040s
I20250626 08:24:16.021164 4996 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250626 08:24:15.588238 5042 log.cc:927] Time spent T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.156s user 0.001s sys 0.000s
W20250626 08:24:15.902923 4932 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 2.154s user 0.011s sys 0.007s
I20250626 08:24:16.021327 4634 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:15.325640 5013 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:15.961954 4954 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.154215 4954 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.156977 4954 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:15.718856 4819 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.158062 4819 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.160645 4819 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.168043 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:16.169339 1846 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1868) took 1529 ms (client timeout 1934 ms). Trace:
I20250626 08:24:16.172022 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2218) took 1210 ms. Trace:
I20250626 08:24:16.172652 2002 rpcz_store.cc:276] 0626 08:24:14.961974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.127125 (+165151us) service_pool.cc:225] Handling call
0626 08:24:15.473024 (+345899us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:15.473776 (+ 752us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.171709 (+697933us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b5400131ad0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:16.171931 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":726000512}
W20250626 08:24:16.169834 1846 rpcz_store.cc:269] 0626 08:24:14.639975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.956806 (+316831us) service_pool.cc:225] Handling call
0626 08:24:16.167790 (+1210984us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5800000f98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:16.169256 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8857344}
W20250626 08:24:16.176096 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 885281us
I20250626 08:24:16.141590 1919 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250626 08:24:16.179494 1968 leader_election.cc:336] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.555s (SENT)
I20250626 08:24:16.180207 1968 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:16.062404 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.181419 1849 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:16.182165 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 534) took 1551 ms. Trace:
I20250626 08:24:16.182606 1849 rpcz_store.cc:276] 0626 08:24:14.630690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.062115 (+1431425us) service_pool.cc:225] Handling call
0626 08:24:16.182093 (+119978us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.184425 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:15.339680 5018 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.332s user 0.000s sys 0.000s
I20250626 08:24:16.185477 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1881) took 1541 ms. Trace:
I20250626 08:24:16.009099 1923 rpcz_store.cc:276] 0626 08:24:14.353444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.357208 (+ 3764us) service_pool.cc:225] Handling call
0626 08:24:14.357489 (+ 281us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.407109 (+ 49620us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.407142 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:14.412289 (+ 5147us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.450164 (+ 37875us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:14.450223 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:15.997816 (+1547593us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b540030f554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:15.997902 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.997989 (+ 87us) raft_consensus.cc:1683] finished
0626 08:24:15.998010 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:15.998333 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15989,"raft.queue_time_us":11643,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":4486,"spinlock_wait_cycles":87354752,"thread_start_us":13570,"threads_started":1,"wal-append.queue_time_us":35551,"wal-append.run_cpu_time_us":2327,"wal-append.run_wall_time_us":1114247}
I20250626 08:24:16.064049 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 532) took 1433 ms. Trace:
I20250626 08:24:16.185823 1846 rpcz_store.cc:276] 0626 08:24:14.644498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.184137 (+1539639us) service_pool.cc:225] Handling call
0626 08:24:16.185416 (+ 1279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.186039 1848 rpcz_store.cc:276] 0626 08:24:14.630316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.134183 (+503867us) service_pool.cc:225] Handling call
0626 08:24:16.063972 (+929789us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.186935 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.186902 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.187891 1848 raft_consensus.cc:2408] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20250626 08:24:15.588016 5032 log.cc:927] Time spent T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.342s user 0.001s sys 0.000s
I20250626 08:24:16.188225 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 529) took 1558 ms. Trace:
I20250626 08:24:16.188647 1846 rpcz_store.cc:276] 0626 08:24:14.629545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.186652 (+1557107us) service_pool.cc:225] Handling call
0626 08:24:16.188173 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:16.188994 5054 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.603s user 0.001s sys 0.002s
I20250626 08:24:16.064659 2586 rpcz_store.cc:276] 0626 08:24:14.119683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.131297 (+ 11614us) service_pool.cc:225] Handling call
0626 08:24:14.131620 (+ 323us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.149415 (+ 17795us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.149446 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:14.164044 (+ 14598us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.278593 (+114549us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:14.278644 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.278716 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.475611 (+1196895us) raft_consensus.cc:1683] finished
0626 08:24:15.475639 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:15.475819 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154354,"raft.queue_time_us":16533,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":12227,"thread_start_us":76128,"threads_started":1,"wal-append.queue_time_us":268459}
I20250626 08:24:16.189520 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.190079 1846 raft_consensus.cc:2374] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6. Re-sending same reply.
I20250626 08:24:16.190207 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2212) took 1638 ms. Trace:
I20250626 08:24:16.023342 1845 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:16.190539 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1880) took 1546 ms. Trace:
W20250626 08:24:16.190798 1845 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 531) took 1560 ms (client timeout 1658 ms). Trace:
I20250626 08:24:16.190886 1846 rpcz_store.cc:276] 0626 08:24:14.644282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.189276 (+1544994us) service_pool.cc:225] Handling call
0626 08:24:16.190490 (+ 1214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:16.191198 1845 rpcz_store.cc:269] 0626 08:24:14.630147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.804758 (+174611us) service_pool.cc:225] Handling call
0626 08:24:16.190743 (+1385985us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.192040 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:15.343119 5037 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.192979 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 530) took 1563 ms. Trace:
I20250626 08:24:15.955713 4496 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.019522 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1803) took 1043 ms. Trace:
I20250626 08:24:16.198304 1999 rpcz_store.cc:276] 0626 08:24:14.961286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.989477 (+ 28191us) service_pool.cc:225] Handling call
0626 08:24:16.005078 (+1015601us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:15.338196 5038 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:15.587291 5027 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.124752 4979 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.020174 1927 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:16.202733 5027 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.203845 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2094) took 1058 ms. Trace:
I20250626 08:24:16.204241 1927 rpcz_store.cc:276] 0626 08:24:15.145869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.007489 (+861620us) service_pool.cc:225] Handling call
0626 08:24:16.203781 (+196292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.205348 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:15.156407 4945 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.865s
W20250626 08:24:15.891042 5031 log.cc:927] Time spent T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.682s user 0.001s sys 0.000s
I20250626 08:24:16.207036 4948 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:16.207705 4948 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:15.336838 5028 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:16.208439 2582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 657) took 1608 ms. Trace:
I20250626 08:24:16.208681 4948 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:15.342859 5011 raft_consensus.cc:2747] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:16.102244 5049 consensus_queue.cc:1046] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.140s
I20250626 08:24:15.344758 4832 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.210601 2584 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250626 08:24:16.210846 5027 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.210886 5049 raft_consensus.cc:3053] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:16.211503 5049 raft_consensus.cc:738] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:16.212141 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.213047 1923 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:16.212422 5049 consensus_queue.cc:260] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.213522 4959 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:16.193485 5013 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.214140 4959 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:15.889710 5021 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.023s sys 0.012s
W20250626 08:24:16.215046 5021 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s user 0.028s sys 0.017s
I20250626 08:24:16.215123 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:16.069340 2585 rpcz_store.cc:269] 0626 08:24:13.586114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:13.586272 (+ 158us) service_pool.cc:225] Handling call
0626 08:24:15.891911 (+2305639us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54005442d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:15.892059 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2528094592}
I20250626 08:24:15.470477 4662 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:16.216440 4662 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.216848 1814 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:16.218490 4662 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:15.236594 5000 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:16.218995 1928 raft_consensus.cc:1238] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
W20250626 08:24:15.583690 5015 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.373s user 0.002s sys 0.000s
I20250626 08:24:16.219609 5000 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.221117 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 577) took 1883 ms. Trace:
W20250626 08:24:15.888978 4997 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.005s sys 0.010s
W20250626 08:24:16.222119 4997 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.214s user 0.011s sys 0.014s
I20250626 08:24:16.084267 1996 rpcz_store.cc:276] 0626 08:24:14.107888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.110209 (+ 2321us) service_pool.cc:225] Handling call
0626 08:24:14.127745 (+ 17536us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.171540 (+ 43795us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.171597 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:14.172278 (+ 681us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.352830 (+180552us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:14.352886 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.352959 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:16.083631 (+1730672us) raft_consensus.cc:1683] finished
0626 08:24:16.083660 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.083782 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28573,"raft.queue_time_us":55162,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":9875,"thread_start_us":146088,"threads_started":1,"wal-append.queue_time_us":616234}
W20250626 08:24:15.157756 4971 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.119s user 0.004s sys 0.017s
I20250626 08:24:16.087963 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:16.221633 1921 rpcz_store.cc:276] 0626 08:24:14.337267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.338147 (+ 880us) service_pool.cc:225] Handling call
0626 08:24:14.362634 (+ 24487us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:14.405147 (+ 42513us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.405185 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:14.418296 (+ 13111us) log.cc:844] Serialized 52 byte log entry
0626 08:24:14.540153 (+121857us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:14.540234 (+ 81us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.540365 (+ 131us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:16.220816 (+1680451us) raft_consensus.cc:1683] finished
0626 08:24:16.220845 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.221039 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8953,"raft.queue_time_us":15656,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":2882,"spinlock_wait_cycles":23296,"thread_start_us":80135,"threads_started":1,"wal-append.queue_time_us":211796}
W20250626 08:24:16.223819 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 135916us
I20250626 08:24:15.889448 2580 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEARNER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:16.224941 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2096) took 1078 ms. Trace:
I20250626 08:24:16.083493 1842 rpcz_store.cc:276] 0626 08:24:14.578112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.578311 (+ 199us) service_pool.cc:225] Handling call
0626 08:24:14.610492 (+ 32181us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.691200 (+ 80708us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.691250 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:14.691854 (+ 604us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.693234 (+ 1380us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:14.693279 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.693369 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:16.082900 (+1389531us) raft_consensus.cc:1683] finished
0626 08:24:16.082928 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.083075 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28479,"raft.queue_time_us":111253,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":7100,"thread_start_us":52618,"threads_started":2,"wal-append.queue_time_us":240045}
I20250626 08:24:16.225675 2580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 729) took 2095 ms. Trace:
I20250626 08:24:16.226272 2580 rpcz_store.cc:276] 0626 08:24:14.130030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.131098 (+ 1068us) service_pool.cc:225] Handling call
0626 08:24:15.889000 (+1757902us) spinlock_profiling.cc:243] Waited 7.28 ms on lock 0x7b58002d3398. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e31dc 00007f4eb60e2d9b 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:15.889212 (+ 212us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.225409 (+336197us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.225619 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16741760}
I20250626 08:24:16.226909 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.228200 1842 raft_consensus.cc:2408] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:16.228646 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1884) took 1505 ms. Trace:
W20250626 08:24:16.095608 4973 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.007s sys 0.016s
I20250626 08:24:16.244529 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2097) took 1064 ms. Trace:
I20250626 08:24:15.587544 5043 consensus_queue.cc:1046] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.692s
I20250626 08:24:16.095657 4946 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.084233 5030 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:16.106559 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 563229us
W20250626 08:24:16.229293 5050 log.cc:927] Time spent T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.662s user 0.004s sys 0.000s
I20250626 08:24:16.101684 4916 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.308720 4916 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:16.112354 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.309573 1925 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:16.310789 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.311559 1925 raft_consensus.cc:2391] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.312322 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.313159 1925 raft_consensus.cc:2391] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.314059 1994 raft_consensus.cc:2391] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 10: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:24:16.096117 4987 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.139s user 0.002s sys 0.023s
I20250626 08:24:16.317422 4987 raft_consensus.cc:695] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:16.318605 4987 consensus_queue.cc:237] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:16.316906 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2213) took 1763 ms (client timeout 2238 ms). Trace:
I20250626 08:24:16.326733 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 660) took 1195 ms. Trace:
I20250626 08:24:16.327276 2584 rpcz_store.cc:276] 0626 08:24:15.131497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.327079 (+195582us) service_pool.cc:225] Handling call
0626 08:24:15.474407 (+147328us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.210976 (+736569us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.326377 (+115401us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540060c554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:16.326617 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1692303744}
I20250626 08:24:15.249992 5047 consensus_queue.cc:1035] T edeafceae4424506a7bffa7a9fbac201 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.966s
I20250626 08:24:15.995472 4840 consensus_queue.cc:260] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.102083 5063 consensus_queue.cc:1046] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20250626 08:24:16.130859 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 644) took 3464 ms. Trace:
I20250626 08:24:16.106621 5019 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.138060 4634 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:16.124013 4932 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 2.372s user 0.012s sys 0.017s
I20250626 08:24:16.113377 4996 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:16.144765 5052 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.393s user 0.002s sys 0.000s
W20250626 08:24:16.148391 5033 log.cc:927] Time spent T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.739s user 0.000s sys 0.001s
I20250626 08:24:16.085199 4895 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.097012 4980 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.184612 4719 raft_consensus.cc:3053] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:15.688678 4928 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:16.191998 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:15.595742 4495 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.195178 1812 leader_election.cc:400] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
I20250626 08:24:16.367543 1812 leader_election.cc:403] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:16.195984 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1935) took 1050 ms. Trace:
I20250626 08:24:16.188275 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 542) took 1556 ms. Trace:
I20250626 08:24:16.161334 4819 consensus_queue.cc:1046] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250626 08:24:16.200131 1968 leader_election.cc:336] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.562s (SENT)
I20250626 08:24:16.200663 5037 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.201032 4940 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:16.200862 4496 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.203263 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2085) took 1567 ms. Trace:
I20250626 08:24:15.337831 5029 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:16.208884 2582 rpcz_store.cc:276] 0626 08:24:14.600471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.768731 (+168260us) service_pool.cc:225] Handling call
0626 08:24:14.769065 (+ 334us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.769402 (+ 337us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.769440 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:14.769974 (+ 534us) log.cc:844] Serialized 26 byte log entry
0626 08:24:14.871359 (+101385us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:14.871526 (+ 167us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.871592 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.890263 (+1018671us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:16.208092 (+317829us) raft_consensus.cc:1683] finished
0626 08:24:16.208122 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.208353 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":155984,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":17395,"spinlock_wait_cycles":42117888,"thread_start_us":83433,"threads_started":1,"wal-append.queue_time_us":266346}
I20250626 08:24:16.375305 1920 rpcz_store.cc:276] 0626 08:24:14.635445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.649025 (+ 13580us) service_pool.cc:225] Handling call
0626 08:24:14.711576 (+ 62551us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:15.135860 (+424284us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.202998 (+1067138us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54001aa254. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:16.203188 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":939726720}
W20250626 08:24:16.070955 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 913532us
W20250626 08:24:16.023219 1893 leader_election.cc:336] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.036s (SENT)
I20250626 08:24:15.277354 4919 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.214238 5013 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:15.695279 4629 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.376s
I20250626 08:24:16.190548 2000 rpcz_store.cc:276] 0626 08:24:14.551933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.552103 (+ 170us) service_pool.cc:225] Handling call
0626 08:24:14.572248 (+ 20145us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:14.583361 (+ 11113us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:14.583410 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:14.584191 (+ 781us) log.cc:844] Serialized 52 byte log entry
0626 08:24:14.885005 (+300814us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:14.927266 (+ 42261us) log.cc:844] Serialized 14 byte log entry
0626 08:24:14.927524 (+ 258us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:14.927624 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.600065 (+672441us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b1800307f00. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb609825a 00007f4ebf55a229 00007f4ebcf3c5cf 00007f4eb612065e 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd
0626 08:24:16.170408 (+570343us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b1800307f00. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:16.189987 (+ 19579us) raft_consensus.cc:1683] finished
0626 08:24:16.190008 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.190151 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6155,"prepare.queue_time_us":43168,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":1135,"raft.run_cpu_time_us":826,"raft.run_wall_time_us":60808,"spinlock_wait_cycles":34231808,"thread_start_us":198600,"threads_started":1,"wal-append.queue_time_us":393017}
I20250626 08:24:16.193440 1845 rpcz_store.cc:276] 0626 08:24:14.629826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.191807 (+1561981us) service_pool.cc:225] Handling call
0626 08:24:16.192906 (+ 1099us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25728}
I20250626 08:24:16.219650 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 581) took 1075 ms. Trace:
I20250626 08:24:16.367071 4840 raft_consensus.cc:3058] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:16.225411 1921 rpcz_store.cc:276] 0626 08:24:15.146504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.224029 (+1077525us) service_pool.cc:225] Handling call
0626 08:24:16.224445 (+ 416us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.224637 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.224666 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.224704 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:16.224742 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.224793 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.224888 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.240873 1927 raft_consensus.cc:2408] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250626 08:24:16.249186 1923 rpcz_store.cc:276] 0626 08:24:15.180321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.232080 (+1051759us) service_pool.cc:225] Handling call
0626 08:24:16.243719 (+ 11639us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.244021 (+ 302us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.244065 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.244132 (+ 67us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:16.244179 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.244260 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.244442 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.253633 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.101778 4917 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.257701 1814 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:16.209801 4948 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:16.260596 2551 leader_election.cc:336] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.866s (SENT)
I20250626 08:24:16.111474 5051 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:16.397962 5051 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.374291 4928 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:16.248147 4973 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.665s user 0.007s sys 0.016s
I20250626 08:24:16.100318 5060 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.137s
I20250626 08:24:16.270802 5030 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.407063 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.407840 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1876) took 1765 ms. Trace:
I20250626 08:24:16.407902 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.408243 1845 rpcz_store.cc:276] 0626 08:24:14.642168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.396314 (+1754146us) service_pool.cc:225] Handling call
0626 08:24:16.407780 (+ 11466us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.271248 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.272512 4946 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.273367 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.274758 2587 raft_consensus.cc:3053] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:16.416361 2587 raft_consensus.cc:738] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:16.276412 1999 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20250626 08:24:16.277616 5072 log.cc:927] Time spent T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20250626 08:24:16.279064 4959 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.279408 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.280300 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.282562 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2200) took 3231 ms. Trace:
W20250626 08:24:16.284433 5071 log.cc:927] Time spent T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20250626 08:24:16.285467 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.286118 4979 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.287365 4988 raft_consensus.cc:3053] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:16.290063 5049 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:16.250324 5021 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:16.254751 5054 log.cc:927] Time spent T 8e262c3543404a9cb840169098735ea2 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20250626 08:24:16.292181 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2209) took 1967 ms. Trace:
I20250626 08:24:16.305205 4807 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:16.331487 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1934) took 1186 ms. Trace:
W20250626 08:24:16.437857 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 422868us
I20250626 08:24:16.523594 1998 rpcz_store.cc:276] 0626 08:24:14.324899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.325367 (+ 468us) service_pool.cc:225] Handling call
0626 08:24:14.342609 (+ 17242us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:15.896630 (+1554021us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5400380ca0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900e2f0 00007f4eb9080da0 00007f4eb9081e11 00007f4eb907e97f 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c
0626 08:24:16.064466 (+167836us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.291887 (+227421us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b5400298954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:16.292108 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1603710080}
I20250626 08:24:16.523407 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:16.461356 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.525698 1918 rpcz_store.cc:276] 0626 08:24:15.144921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.145121 (+ 200us) service_pool.cc:225] Handling call
0626 08:24:16.330328 (+1185207us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e31dc 00007f4eb60e2d9b 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:16.330580 (+ 252us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.331035 (+ 455us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.331076 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.331136 (+ 60us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:16.331181 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.331258 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.331418 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4782848}
W20250626 08:24:16.399842 4966 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 2.251s user 0.033s sys 0.029s
I20250626 08:24:16.495820 5075 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
W20250626 08:24:16.528785 4966 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 2.380s user 0.033s sys 0.033s
W20250626 08:24:16.495592 5017 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.004s sys 0.002s
I20250626 08:24:16.462610 2587 consensus_queue.cc:260] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.267576 4999 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:16.497535 4954 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:16.530061 4999 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:16.524276 5021 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:16.530683 4999 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:16.530922 5021 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:16.531392 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.532169 1918 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.361367 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:16.364089 1847 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1792) took 2905 ms (client timeout 2926 ms). Trace:
I20250626 08:24:16.498291 5044 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:16.534217 1847 rpcz_store.cc:269] 0626 08:24:13.459247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.522305 (+1063058us) service_pool.cc:225] Handling call
0626 08:24:16.363879 (+1841574us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54002bacd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:16.364028 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3535110912}
I20250626 08:24:16.534621 5044 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.535336 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.535948 1847 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:16.536406 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1888) took 1809 ms. Trace:
I20250626 08:24:16.536712 1847 rpcz_store.cc:276] 0626 08:24:14.727040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.535087 (+1808047us) service_pool.cc:225] Handling call
0626 08:24:16.536356 (+ 1269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.537117 5044 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:16.537660 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:16.365324 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:16.356904 4955 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.003s sys 0.052s
W20250626 08:24:16.369113 1846 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1872) took 1727 ms (client timeout 2249 ms). Trace:
I20250626 08:24:16.369586 2584 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:16.509333 4634 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:16.542110 1846 rpcz_store.cc:269] 0626 08:24:14.641214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.191755 (+1550541us) service_pool.cc:225] Handling call
0626 08:24:16.369050 (+177295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.368918 1848 rpcz_store.cc:276] 0626 08:24:14.631919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.186764 (+1554845us) service_pool.cc:225] Handling call
0626 08:24:16.188229 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.543018 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.543339 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.543598 1846 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:16.544107 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1892) took 1815 ms. Trace:
I20250626 08:24:16.544178 1848 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 10 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250626 08:24:16.544407 1846 rpcz_store.cc:276] 0626 08:24:14.728301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.542783 (+1814482us) service_pool.cc:225] Handling call
0626 08:24:16.544058 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.544680 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1894) took 1815 ms. Trace:
I20250626 08:24:16.340826 2583 rpcz_store.cc:276] 0626 08:24:12.642025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.644492 (+ 2467us) service_pool.cc:225] Handling call
0626 08:24:12.653998 (+ 9506us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:12.941071 (+287073us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:12.974631 (+ 33560us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:13.034346 (+ 59715us) log.cc:844] Serialized 26 byte log entry
0626 08:24:13.546807 (+512461us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:13.546879 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:13.548046 (+ 1167us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.790509 (+2242463us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:16.084248 (+293739us) raft_consensus.cc:1683] finished
0626 08:24:16.084273 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.106490 (+ 22217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":190676,"raft.queue_time_us":216407,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":131,"spinlock_wait_cycles":97803008,"thread_start_us":557933,"threads_started":2,"wal-append.queue_time_us":871612}
I20250626 08:24:16.545296 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.523411 5013 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:16.546221 4955 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.135s user 0.004s sys 0.052s
I20250626 08:24:16.547809 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1856) took 1915 ms. Trace:
I20250626 08:24:16.499687 5037 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.548306 1846 rpcz_store.cc:276] 0626 08:24:14.632085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.545073 (+1912988us) service_pool.cc:225] Handling call
0626 08:24:16.547522 (+ 2449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.549247 4895 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.549418 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.497123 4496 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.550371 4496 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:16.550426 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1893) took 1821 ms. Trace:
I20250626 08:24:16.550987 1846 rpcz_store.cc:276] 0626 08:24:14.728604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.549134 (+1820530us) service_pool.cc:225] Handling call
0626 08:24:16.550363 (+ 1229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.550995 4496 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.551977 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.552698 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1886) took 1826 ms. Trace:
I20250626 08:24:16.552982 1846 rpcz_store.cc:276] 0626 08:24:14.726407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.551734 (+1825327us) service_pool.cc:225] Handling call
0626 08:24:16.552643 (+ 909us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:16.323436 1994 rpcz_store.cc:269] 0626 08:24:14.553214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.553377 (+ 163us) service_pool.cc:225] Handling call
0626 08:24:16.316667 (+1763290us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54000f5854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:16.316832 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4017641728}
I20250626 08:24:16.357188 4996 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Advancing to term 10
W20250626 08:24:16.373469 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 173373us
W20250626 08:24:16.482636 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 225041us
W20250626 08:24:16.573770 1968 leader_election.cc:336] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.610s (SENT)
I20250626 08:24:16.573330 1965 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:16.576741 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:24:16.402936 5004 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.166s user 0.006s sys 0.009s
I20250626 08:24:16.382735 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:16.584072 2582 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:16.387107 1928 rpcz_store.cc:276] 0626 08:24:15.144333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.144455 (+ 122us) service_pool.cc:225] Handling call
0626 08:24:16.218643 (+1074188us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e31dc 00007f4eb60e2d9b 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:16.218827 (+ 184us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.219401 (+ 574us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.219583 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37599360}
I20250626 08:24:16.229030 1842 rpcz_store.cc:276] 0626 08:24:14.723369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.226520 (+1503151us) service_pool.cc:225] Handling call
0626 08:24:16.228565 (+ 2045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.366458 4719 raft_consensus.cc:738] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:16.395593 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:16.602454 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.394528 2578 raft_consensus.cc:1238] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:16.398437 4980 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:16.583737 5004 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.375s user 0.008s sys 0.015s
I20250626 08:24:16.612404 1965 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:16.400403 1926 raft_consensus.cc:1238] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:16.400480 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:16.264961 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.404920 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:16.368479 4916 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250626 08:24:16.409268 2002 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:16.615130 1927 raft_consensus.cc:2391] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 8: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.367538 5055 consensus_queue.cc:1035] T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.030s
I20250626 08:24:16.373898 4495 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.263041 5025 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.624387 5025 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:16.529488 5017 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.005s sys 0.003s
I20250626 08:24:16.624553 4719 consensus_queue.cc:260] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.605933 4634 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:16.583443 4819 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:16.510512 5019 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:16.511616 4928 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.545336 1847 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:16.630962 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1882) took 1986 ms. Trace:
W20250626 08:24:16.438105 1849 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1871) took 1797 ms (client timeout 2195 ms). Trace:
I20250626 08:24:16.629408 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:16.631752 1849 rpcz_store.cc:269] 0626 08:24:14.640981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.197200 (+1556219us) service_pool.cc:225] Handling call
0626 08:24:16.438031 (+240831us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.545069 1848 rpcz_store.cc:276] 0626 08:24:14.728902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.543117 (+1814215us) service_pool.cc:225] Handling call
0626 08:24:16.544625 (+ 1508us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.553793 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:16.369838 1919 rpcz_store.cc:276] 0626 08:24:15.145509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:15.145695 (+ 186us) service_pool.cc:225] Handling call
0626 08:24:16.140957 (+995262us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b5800091e98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e322e 0000000000356d09 00007f4ebdf5bdc0 00007f4ebe0227d4 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:16.141177 (+ 220us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.178850 (+ 37673us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.195908 (+ 17058us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13485056}
I20250626 08:24:16.632931 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:16.632982 1847 rpcz_store.cc:276] 0626 08:24:14.644714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.537356 (+1892642us) service_pool.cc:225] Handling call
0626 08:24:16.630902 (+ 93546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.633639 1927 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:16.558670 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 381755us
I20250626 08:24:16.634552 1812 leader_election.cc:400] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7.
I20250626 08:24:16.634845 1812 leader_election.cc:403] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:16.382570 1924 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
W20250626 08:24:16.601287 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.601732 4496 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.566813 1993 rpcz_store.cc:276] 0626 08:24:13.051190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:13.051495 (+ 305us) service_pool.cc:225] Handling call
0626 08:24:13.104792 (+ 53297us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:13.410864 (+306072us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:24:13.587802 (+176938us) log.cc:844] Serialized 14 byte log entry
0626 08:24:13.772290 (+184488us) log.cc:844] Serialized 14 byte log entry
0626 08:24:13.772618 (+ 328us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:13.792199 (+ 19581us) log.cc:844] Serialized 52 byte log entry
0626 08:24:13.869696 (+ 77497us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:13.872570 (+ 2874us) log.cc:844] Serialized 14 byte log entry
0626 08:24:13.875172 (+ 2602us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:13.875738 (+ 566us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:15.691910 (+1816172us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:16.142200 (+450290us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b180009cd00. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:16.282341 (+140141us) raft_consensus.cc:1683] finished
0626 08:24:16.282360 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.282505 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126642,"prepare.queue_time_us":5017373,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":416359,"raft.run_cpu_time_us":698,"raft.run_wall_time_us":82111,"spinlock_wait_cycles":74235008,"thread_start_us":282156,"threads_started":2,"wal-append.queue_time_us":326882}
I20250626 08:24:16.400911 5027 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.606495 5037 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:16.523192 4917 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:16.607293 2587 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:16.416481 5006 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.608919 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.495349 5084 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:16.614836 1920 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:16.685001 1928 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.610515 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:16.621111 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.622190 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.573771 4988 raft_consensus.cc:738] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:16.529717 1845 raft_consensus.cc:2391] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
W20250626 08:24:16.628322 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:16.643085 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 118595us
W20250626 08:24:16.693441 1893 connection.cc:232] RPC call timeout handler was delayed by -2.0213461790003748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2396s for a response
W20250626 08:24:16.643471 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.644762 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1889) took 1917 ms. Trace:
I20250626 08:24:16.694977 1846 rpcz_store.cc:276] 0626 08:24:14.727342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.553560 (+1826218us) service_pool.cc:225] Handling call
0626 08:24:16.644698 (+ 91138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.696453 1846 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250626 08:24:16.697134 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1896) took 1967 ms. Trace:
I20250626 08:24:16.587644 2585 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
W20250626 08:24:16.648089 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 387513us
I20250626 08:24:16.660233 1919 raft_consensus.cc:1238] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:16.621946 4979 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:16.707311 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:16.709220 1849 raft_consensus.cc:1273] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250626 08:24:16.709599 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 93302.5us
I20250626 08:24:16.710021 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1860) took 2077 ms. Trace:
I20250626 08:24:16.710527 1849 rpcz_store.cc:276] 0626 08:24:14.632658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.644967 (+2012309us) service_pool.cc:225] Handling call
0626 08:24:16.704981 (+ 60014us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.709671 (+ 4690us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.709941 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:16.708194 2581 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 3.
I20250626 08:24:16.628935 5086 raft_consensus.cc:2747] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:16.724218 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.648823 5037 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:16.619151 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:16.619927 5004 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:16.746371 1921 raft_consensus.cc:2391] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:16.694528 1967 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:16.701901 1893 connection.cc:232] RPC call timeout handler was delayed by -1.8548461790001056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2581s for a response
I20250626 08:24:16.627631 5051 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.697510 1846 rpcz_store.cc:276] 0626 08:24:14.729502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.695793 (+1966291us) service_pool.cc:225] Handling call
0626 08:24:16.696177 (+ 384us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:16.696846 (+ 669us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.697069 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:16.747566 1893 connection.cc:232] RPC call timeout handler was delayed by -1.5252344290001929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33370000000000005s for a response
W20250626 08:24:16.748121 1893 connection.cc:232] RPC call timeout handler was delayed by -1.4489461790003588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3032s for a response
W20250626 08:24:16.748428 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3832461790002526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3105s for a response
I20250626 08:24:16.749094 4946 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:16.749593 4946 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.751955 4946 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.752432 4946 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:16.752831 4946 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:16.754571 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1528461790003348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33610000000000007s for a response
I20250626 08:24:16.737422 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:16.755472 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1050804930000595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
I20250626 08:24:16.739347 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.756330 1997 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 9.
W20250626 08:24:16.756644 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 122211us
I20250626 08:24:16.743701 1919 raft_consensus.cc:1273] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20250626 08:24:16.704537 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.744084 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.767457 1993 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:16.626480 5044 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20250626 08:24:16.628674 1965 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:16.769203 1998 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:16.691390 5084 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:16.780267 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:16.780669 1928 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:16.757301 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:16.771189 1927 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:16.785157 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.785766 1993 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:24:16.786541 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 92101.4us
I20250626 08:24:16.646596 5019 consensus_queue.cc:1046] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.097s
I20250626 08:24:16.644304 4819 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:16.773662 2551 leader_election.cc:336] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.089s (SENT)
I20250626 08:24:16.789656 2551 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:16.723003 4629 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7.
I20250626 08:24:16.802204 1919 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:16.804491 1920 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250626 08:24:16.754530 5037 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:16.684259 5027 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.705439 5006 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.660073 4959 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20250626 08:24:16.841590 2582 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:16.841945 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:16.739710 5030 consensus_queue.cc:1046] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
W20250626 08:24:16.846019 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 217434us
I20250626 08:24:16.848444 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:16.854162 1919 raft_consensus.cc:1238] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:16.799467 4496 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:16.858807 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.711845 5021 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:16.863794 1996 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:16.809831 4955 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20250626 08:24:16.789978 5084 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:16.867097 1920 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:16.800252 4917 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:16.812158 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 656) took 2317 ms. Trace:
I20250626 08:24:16.807914 4662 consensus_queue.cc:1035] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250626 08:24:16.863991 5044 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:16.890154 2583 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:16.863016 4987 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:16.863930 4496 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:16.852648 4895 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20250626 08:24:16.921316 4946 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.923689 5049 raft_consensus.cc:3053] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:16.924226 2586 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:16.898555 4496 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:16.864141 4956 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20250626 08:24:16.927548 4840 consensus_meta.cc:287] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.002s
W20250626 08:24:16.898209 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 323560us
W20250626 08:24:16.933593 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45848.3us
W20250626 08:24:16.937623 5085 log.cc:927] Time spent T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20250626 08:24:16.914188 4495 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:16.931967 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:16.958520 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:16.963680 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 225540us
I20250626 08:24:16.886725 5004 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:16.937719 5049 raft_consensus.cc:738] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:17.051697 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 103321us
I20250626 08:24:17.048889 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:16.945385 5041 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:17.062908 2587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 740) took 2297 ms. Trace:
I20250626 08:24:17.063064 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:17.065140 2585 raft_consensus.cc:2408] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20250626 08:24:17.064235 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 251862us
I20250626 08:24:17.066537 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:17.068554 1968 leader_election.cc:336] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 3.124s (SENT)
I20250626 08:24:17.069160 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:17.069080 1816 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:17.077646 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:17.110564 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 353299us
I20250626 08:24:17.065542 2587 rpcz_store.cc:276] 0626 08:24:14.765816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.905390 (+139574us) service_pool.cc:225] Handling call
0626 08:24:14.905764 (+ 374us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.274606 (+1368842us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b540065caa0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb900e2f0 00007f4eb9080da0 00007f4eb9081e11 00007f4eb907e97f 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c
0626 08:24:17.062561 (+787955us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:17.062591 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:17.062631 (+ 40us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:17.062665 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.062736 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:17.062840 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":173423,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":162077952,"thread_start_us":291876,"threads_started":1}
I20250626 08:24:17.112288 1812 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.112644 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:17.114444 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:17.115443 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 46574.9us
W20250626 08:24:17.117870 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:17.120795 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 61885.9us
I20250626 08:24:17.122813 1893 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:24:17.120332 4486 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
W20250626 08:24:17.159906 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 91395us
W20250626 08:24:17.164005 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:17.185870 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.121243 4728 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250626 08:24:17.218524 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 106311us
I20250626 08:24:17.219630 1812 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.187160 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1883) took 2509 ms. Trace:
I20250626 08:24:17.113973 5024 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:17.122655 5030 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250626 08:24:17.123845 4948 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250626 08:24:17.181952 4999 consensus_meta.cc:287] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.651s user 0.009s sys 0.002s
W20250626 08:24:17.229456 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 353175us
I20250626 08:24:17.159821 4819 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.135728 5027 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:17.159595 4959 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:17.122035 5017 consensus_queue.cc:1046] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20250626 08:24:17.111258 5041 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.224078 5030 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:17.237834 4662 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:17.254549 4662 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.257680 4662 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.222200 1848 rpcz_store.cc:276] 0626 08:24:14.659249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.644369 (+1985120us) service_pool.cc:225] Handling call
0626 08:24:16.644691 (+ 322us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.709208 (+ 64517us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:24:16.768912 (+ 59704us) log.cc:844] Serialized 14 byte log entry
0626 08:24:17.153687 (+384775us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:17.153785 (+ 98us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:17.153858 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.153975 (+ 117us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:17.154165 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76079,"prepare.queue_time_us":46486,"raft.queue_time_us":64575,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":22158,"spinlock_wait_cycles":23714560,"thread_start_us":277178,"threads_started":1}
W20250626 08:24:17.285374 4634 consensus_meta.cc:287] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.000s
I20250626 08:24:17.250154 4999 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.285872 4634 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:17.229493 5099 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:17.240216 4840 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:17.300837 1850 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:17.293769 1965 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.239521 5021 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.240006 5006 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.247449 4495 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:17.289865 4962 log.cc:927] Time spent T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250626 08:24:17.339457 1999 raft_consensus.cc:1238] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:17.252978 4819 raft_consensus.cc:2747] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:17.287650 4999 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.288311 5030 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:17.350349 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:17.301316 5099 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:17.389391 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.333053 4634 raft_consensus.cc:2747] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250626 08:24:17.346312 4807 consensus_meta.cc:287] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.804s user 0.005s sys 0.005s
I20250626 08:24:17.234880 5024 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.400941 1993 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250626 08:24:17.414547 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:17.495083 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.250473 5049 consensus_queue.cc:260] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.519659 1844 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
W20250626 08:24:17.520430 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 746747us
W20250626 08:24:17.522356 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:16.872238 2579 rpcz_store.cc:276] 0626 08:24:14.491139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.774883 (+283744us) service_pool.cc:225] Handling call
0626 08:24:14.825819 (+ 50936us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:15.274746 (+448927us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:15.283902 (+ 9156us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:15.337138 (+ 53236us) log.cc:844] Serialized 52 byte log entry
0626 08:24:15.893497 (+556359us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:15.893576 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:15.893706 (+ 130us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:16.283773 (+390067us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:16.723684 (+439911us) raft_consensus.cc:1683] finished
0626 08:24:16.738457 (+ 14773us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:16.808766 (+ 70309us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27698,"raft.queue_time_us":72193,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10387,"spinlock_wait_cycles":42071296,"thread_start_us":753517,"threads_started":2,"wal-append.queue_time_us":745713}
I20250626 08:24:17.391438 5099 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:17.397449 5017 raft_consensus.cc:3053] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:17.351738 4495 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.566947 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:17.567790 1993 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:17.561059 5098 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:17.574869 5098 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.532888 4840 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:17.561168 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.552966 4948 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:17.586969 5024 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.587986 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:17.588384 5044 consensus_meta.cc:287] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.690s user 0.009s sys 0.000s
I20250626 08:24:17.588820 5044 raft_consensus.cc:2747] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:17.579669 5102 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:17.590273 5102 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.591217 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:17.592175 5102 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.592489 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:17.593112 1921 raft_consensus.cc:3058] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:17.594391 4643 consensus_queue.cc:1046] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:24:17.596135 4948 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:17.584229 2551 leader_election.cc:336] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.218s (SENT)
I20250626 08:24:17.599923 2551 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:24:17.600677 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 477902us
I20250626 08:24:17.602648 1893 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:17.602969 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:17.563756 5017 raft_consensus.cc:738] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:17.610980 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.631765 2551 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:17.633764 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:17.636544 2000 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:17.629040 2581 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.015s sys 0.020s
I20250626 08:24:17.637797 2585 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cca900160913464799d0cfa9d2866779"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:45504
I20250626 08:24:17.638383 2585 raft_consensus.cc:604] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Received request to transfer leadership to TS 755d83ca0913406b816e95415d502055
I20250626 08:24:17.638751 2585 raft_consensus.cc:612] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:17.638530 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:17.605755 4996 consensus_meta.cc:287] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.001s user 0.008s sys 0.004s
I20250626 08:24:17.641330 1893 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.648605 1995 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:17.650077 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:17.651777 1893 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
W20250626 08:24:17.684677 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 465125us
I20250626 08:24:17.598098 4840 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.683703 5004 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:17.708499 5004 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:17.709040 5004 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:17.712495 4943 raft_consensus.cc:668] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } } [suppressed 3 similar messages]
I20250626 08:24:17.719672 4943 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:17.722153 4943 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:17.724609 4943 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.731876 4943 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:17.732679 4943 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250626 08:24:17.734169 5017 consensus_queue.cc:260] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.735862 5017 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:17.737061 4943 raft_consensus.cc:2762] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250626 08:24:17.746435 5113 raft_consensus.cc:2747] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:17.762431 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:17.771561 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 85803.4us
I20250626 08:24:17.689980 4948 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.787874 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2103) took 1150 ms. Trace:
I20250626 08:24:17.788261 1928 rpcz_store.cc:276] 0626 08:24:16.637455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.838445 (+200990us) service_pool.cc:225] Handling call
0626 08:24:16.838868 (+ 423us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.898869 (+ 60001us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.898908 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.898970 (+ 62us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:16.899015 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.787661 (+888646us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b54002ae9d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:17.787703 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:17.787819 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":137512320}
I20250626 08:24:17.794353 2000 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:24:17.797441 5037 consensus_meta.cc:287] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.942s user 0.007s sys 0.004s
W20250626 08:24:17.802433 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 517) took 3356 ms (client timeout 2089 ms). Trace:
I20250626 08:24:17.808209 5037 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.807341 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:17.809314 1928 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:17.810319 5037 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.680217 4996 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:17.808220 1850 rpcz_store.cc:269] 0626 08:24:14.430942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.431084 (+ 142us) service_pool.cc:225] Handling call
0626 08:24:17.787128 (+3356044us) spinlock_profiling.cc:243] Waited 2.84 s on lock 0x7b54002f1cd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:17.787254 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6521605376}
I20250626 08:24:17.811862 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:17.812408 4996 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:17.812606 1928 raft_consensus.cc:2391] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:17.804419 4948 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:17.813513 4948 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:17.813890 4948 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:17.813980 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:17.814942 1928 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:17.816248 1928 raft_consensus.cc:3053] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:17.816665 1928 raft_consensus.cc:738] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:17.810560 1967 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.775875 5044 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:17.817853 5044 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.820109 5098 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:17.800459 4979 consensus_meta.cc:287] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.069s user 0.012s sys 0.000s
W20250626 08:24:17.842236 1997 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.058s sys 0.143s
I20250626 08:24:17.817478 1928 consensus_queue.cc:260] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.780748 5106 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:17.850078 1968 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.008s sys 0.008s
W20250626 08:24:17.850586 1968 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.008s sys 0.008s
W20250626 08:24:17.851612 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 78071us
W20250626 08:24:17.842629 1997 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.058s sys 0.143s
I20250626 08:24:17.844242 1921 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
W20250626 08:24:17.844450 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 266927us
I20250626 08:24:17.854363 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:17.883488 1921 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
W20250626 08:24:17.884009 1968 leader_election.cc:336] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.770s (SENT)
I20250626 08:24:17.820127 5044 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:17.885011 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.893754 1814 leader_election.cc:304] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:17.843901 4979 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:17.886091 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:17.924767 2586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 663) took 1710 ms (client timeout 2208 ms). Trace:
W20250626 08:24:17.925171 2586 rpcz_store.cc:269] 0626 08:24:16.214634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.214815 (+ 181us) service_pool.cc:225] Handling call
0626 08:24:17.924587 (+1709772us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54006149d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:17.924711 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628781952}
I20250626 08:24:17.926724 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 442) took 1855 ms. Trace:
W20250626 08:24:17.849731 4496 consensus_meta.cc:287] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.900s user 0.004s sys 0.008s
I20250626 08:24:17.886621 1812 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.927559 4496 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.928932 1812 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:17.929257 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:17.887128 1928 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:17.845971 4943 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:17.903561 4807 raft_consensus.cc:2802] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:17.846503 5004 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.904124 4996 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250626 08:24:17.913319 5084 consensus_meta.cc:287] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.020s user 0.008s sys 0.002s
I20250626 08:24:17.847270 5024 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:17.884855 1968 leader_election.cc:336] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.816s (SENT)
I20250626 08:24:17.842351 5116 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:17.921274 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:17.952435 1918 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 6
I20250626 08:24:17.895112 5037 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20250626 08:24:17.930133 1994 raft_consensus.cc:1238] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:17.931061 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:17.954222 1999 raft_consensus.cc:2408] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250626 08:24:17.927098 2001 rpcz_store.cc:276] 0626 08:24:16.071194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.071351 (+ 157us) service_pool.cc:225] Handling call
0626 08:24:16.077611 (+ 6260us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.331002 (+253391us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.331049 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.331110 (+ 61us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:16.331156 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.926484 (+1595328us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54000d0cd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:17.926535 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:17.926670 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582222592}
I20250626 08:24:17.856850 4997 raft_consensus.cc:2747] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:17.940492 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.943508 5099 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.962399 4938 raft_consensus.cc:3053] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:17.962814 4938 raft_consensus.cc:738] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:17.964457 5099 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:17.947710 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 62881.9us
I20250626 08:24:17.949812 1812 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.966753 1812 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.967973 1812 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.933718 4495 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:17.968688 5102 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:17.969067 4960 raft_consensus.cc:3053] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:17.969328 5102 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:17.969450 4960 raft_consensus.cc:738] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:17.970212 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2105) took 1327 ms. Trace:
I20250626 08:24:17.970299 4960 consensus_queue.cc:260] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.929641 4496 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.971670 4960 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:17.971832 5102 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:17.971254 1812 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.973491 5102 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:17.898993 5044 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:17.974191 5044 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.975288 5004 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.975065 1812 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:17.933882 4943 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:17.976675 4943 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:17.977044 1812 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:17.944918 5098 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:17.978449 4496 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:17.978915 5098 raft_consensus.cc:2747] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:17.979008 4496 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:17.979624 5098 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:17.980026 5098 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:17.980368 5098 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:17.981546 4496 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.981770 4948 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:17.982340 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36150 (request call id 702) took 1373 ms. Trace:
I20250626 08:24:17.947686 5084 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:17.983402 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.984117 5024 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: NON_VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: true } }
I20250626 08:24:17.985001 5041 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:17.945127 5017 raft_consensus.cc:2762] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250626 08:24:17.970683 1919 rpcz_store.cc:276] 0626 08:24:16.642448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.871013 (+228565us) service_pool.cc:225] Handling call
0626 08:24:16.897177 (+ 26164us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:16.897429 (+ 252us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.897466 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:16.901167 (+ 3701us) log.cc:844] Serialized 26 byte log entry
0626 08:24:17.231761 (+330594us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:17.231817 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.231915 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:17.969992 (+738077us) raft_consensus.cc:1683] finished
0626 08:24:17.970012 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:17.970143 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":288169,"threads_started":1,"wal-append.queue_time_us":927161}
I20250626 08:24:17.976202 5044 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:17.952431 5116 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:17.945241 5075 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:18.002997 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.022934 1999 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:17.933748 5123 raft_consensus.cc:2747] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:17.965240 4988 consensus_queue.cc:260] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:18.007867 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 356072us
W20250626 08:24:18.010293 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:17.993698 2583 rpcz_store.cc:276] 0626 08:24:16.609048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.611464 (+ 2416us) service_pool.cc:225] Handling call
0626 08:24:16.623539 (+ 12075us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.892181 (+268642us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.982117 (+1089936us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540013bd54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:17.982282 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":612479872}
I20250626 08:24:17.949501 4807 raft_consensus.cc:695] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:18.012413 4832 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.517s user 0.011s sys 0.030s
W20250626 08:24:18.012787 1998 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.008s sys 0.010s
W20250626 08:24:18.043045 1998 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s user 0.008s sys 0.010s
I20250626 08:24:18.043121 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:17.884287 5106 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:18.031353 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.007079 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 357074us
I20250626 08:24:18.034287 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.035548 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:18.047945 1999 raft_consensus.cc:2391] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 10: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:18.048115 1919 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 8: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
W20250626 08:24:18.040084 1842 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.916s user 0.014s sys 0.017s
I20250626 08:24:18.041162 1893 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:24:18.049846 2551 leader_election.cc:336] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.658s (SENT)
W20250626 08:24:18.051169 2551 leader_election.cc:336] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.696s (SENT)
I20250626 08:24:18.011471 5095 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.051635 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.023813 5116 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Advancing to term 9
W20250626 08:24:18.042848 4832 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.547s user 0.011s sys 0.030s
I20250626 08:24:18.060606 1993 raft_consensus.cc:2466] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
W20250626 08:24:18.070726 2551 leader_election.cc:336] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 3.177s (SENT)
I20250626 08:24:18.071237 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.073719 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.082518 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:18.078926 2551 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:18.054100 4988 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:18.087143 1816 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:18.062152 5002 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 08:24:18.070482 5004 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:18.072834 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.073348 4486 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.094050 687 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250626 08:24:18.093029 5017 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:18.102653 5017 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:18.103077 5017 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:18.103808 687 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 08:24:18.104919 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250626 08:24:18.105022 1845 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 541) took 3473 ms (client timeout 3177 ms). Trace:
W20250626 08:24:18.105405 1845 rpcz_store.cc:269] 0626 08:24:14.631773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.408916 (+1777143us) service_pool.cc:225] Handling call
0626 08:24:18.104781 (+1695865us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b540038c054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:18.104935 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5065472}
I20250626 08:24:18.107329 4948 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.107820 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 732) took 3871 ms. Trace:
I20250626 08:24:18.108584 1847 raft_consensus.cc:1273] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250626 08:24:18.112965 1843 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:18.114460 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 447) took 1356 ms. Trace:
I20250626 08:24:18.037823 5075 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.117594 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:18.115382 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1936) took 2820 ms. Trace:
I20250626 08:24:18.121276 1997 rpcz_store.cc:276] 0626 08:24:16.757644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.758110 (+ 466us) service_pool.cc:225] Handling call
0626 08:24:16.784677 (+ 26567us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:16.810894 (+ 26217us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:17.061086 (+250192us) log.cc:844] Serialized 14 byte log entry
0626 08:24:17.856523 (+795437us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:17.858402 (+ 1879us) log.cc:844] Serialized 52 byte log entry
0626 08:24:17.885573 (+ 27171us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:17.886008 (+ 435us) log.cc:844] Serialized 14 byte log entry
0626 08:24:17.902369 (+ 16361us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.902503 (+ 134us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:18.114224 (+211721us) raft_consensus.cc:1683] finished
0626 08:24:18.114249 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.114393 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34003,"prepare.queue_time_us":506071,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":12019,"raft.queue_time_us":99196,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":10156,"spinlock_wait_cycles":39203328,"thread_start_us":831624,"threads_started":2,"wal-append.queue_time_us":928045}
I20250626 08:24:18.117067 2580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 674) took 1146 ms. Trace:
I20250626 08:24:18.090463 5063 raft_consensus.cc:2802] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:18.126376 1923 rpcz_store.cc:276] 0626 08:24:15.295421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.392555 (+1097134us) service_pool.cc:225] Handling call
0626 08:24:16.401750 (+ 9195us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.401992 (+ 242us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.402030 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:16.402088 (+ 58us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:16.402132 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.402201 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.115230 (+1713029us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x7b5400300a50. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:18.115328 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20229376}
I20250626 08:24:18.090024 5114 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:18.126668 2580 rpcz_store.cc:276] 0626 08:24:16.970437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.970603 (+ 166us) service_pool.cc:225] Handling call
0626 08:24:17.062703 (+ 92100us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:17.377111 (+314408us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:17.377156 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:17.400829 (+ 23673us) log.cc:844] Serialized 26 byte log entry
0626 08:24:18.008963 (+608134us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:18.009022 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.009116 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:18.116872 (+107756us) raft_consensus.cc:1683] finished
0626 08:24:18.116890 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.117013 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":468041,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"thread_start_us":675401,"threads_started":2,"wal-append.queue_time_us":559605}
I20250626 08:24:18.133993 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:18.143602 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 161214us
W20250626 08:24:18.123548 5065 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.625s user 0.019s sys 0.033s
I20250626 08:24:18.143813 1923 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:18.147320 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:18.148811 1968 leader_election.cc:336] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.696s (SENT)
W20250626 08:24:18.154824 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.157142 5024 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:18.122354 4807 consensus_queue.cc:237] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.157289 4938 consensus_queue.cc:260] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.142303 4997 raft_consensus.cc:2762] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20250626 08:24:18.160454 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 96911.5us
I20250626 08:24:18.191385 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:18.163046 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:18.166203 1968 leader_election.cc:336] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.830s (SENT)
I20250626 08:24:18.192147 2583 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:18.192381 1995 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:18.192377 1968 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.171958 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.173960 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:18.194090 1968 leader_election.cc:336] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.008s (SENT)
I20250626 08:24:18.196542 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.197762 5123 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250626 08:24:18.120576 2578 rpcz_store.cc:276] 0626 08:24:14.236431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.312019 (+ 75588us) service_pool.cc:225] Handling call
0626 08:24:14.312342 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.619744 (+2307402us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.107541 (+1487797us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54005eae54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:18.107752 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4770609408}
I20250626 08:24:18.200136 5024 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.200836 4495 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:18.201277 4495 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:18.201651 4495 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.205972 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:18.206995 1923 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20250626 08:24:18.207983 2583 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
W20250626 08:24:18.165725 5065 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.649s user 0.020s sys 0.033s
I20250626 08:24:18.209894 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:18.211269 1919 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.213698 1893 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:18.216856 4948 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 08:24:18.218446 5106 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:18.217096 4997 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:18.219228 4997 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:18.219662 4997 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.207418 5134 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.226438 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.227754 5075 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.230666 5132 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:18.231194 5132 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.231617 1965 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.233409 4943 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.233927 5024 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:18.236124 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.236812 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.237954 1893 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.243525 1928 raft_consensus.cc:1273] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250626 08:24:18.244243 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2109) took 1114 ms. Trace:
I20250626 08:24:18.244593 1928 rpcz_store.cc:276] 0626 08:24:17.129896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:17.815843 (+685947us) service_pool.cc:225] Handling call
0626 08:24:17.816141 (+ 298us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.244024 (+427883us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.244195 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:18.170467 5037 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:18.248327 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:18.249334 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.249933 1848 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:18.253796 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.255712 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.256505 1846 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
W20250626 08:24:18.210361 5030 consensus_meta.cc:287] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.813s user 0.010s sys 0.001s
I20250626 08:24:18.258955 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.259109 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.273669 1846 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
W20250626 08:24:18.274385 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 77929.7us
I20250626 08:24:18.274693 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.271710 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:18.275807 1928 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:24:18.277786 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.278565 1998 raft_consensus.cc:3058] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.280047 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.283993 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:18.290402 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.291335 1850 raft_consensus.cc:2374] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7. Re-sending same reply.
I20250626 08:24:18.234477 5024 raft_consensus.cc:695] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:18.299244 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 61364.9us
I20250626 08:24:18.302532 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.299563 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1953) took 1015 ms. Trace:
I20250626 08:24:18.303524 1928 rpcz_store.cc:276] 0626 08:24:17.284516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.276425 (+991909us) service_pool.cc:225] Handling call
0626 08:24:18.299076 (+ 22651us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.299247 (+ 171us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:18.299271 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:18.299314 (+ 43us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:18.299345 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.299399 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.299511 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:18.300262 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.305112 2000 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:18.231756 5084 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:18.300623 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.306174 1968 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.321197 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.322177 1919 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:18.322863 1849 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250626 08:24:18.311134 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 62547.6us
W20250626 08:24:18.330785 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.235778 4486 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:18.332263 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.332903 1850 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:18.333726 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.334457 1850 raft_consensus.cc:2391] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 10: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:18.235435 5063 raft_consensus.cc:695] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:18.335773 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.336388 2582 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:18.337061 1814 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:18.339558 1812 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:18.339818 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.341099 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.341861 1850 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:18.258673 4948 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250626 08:24:18.344156 2551 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:18.347690 1812 leader_election.cc:304] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.294376 5029 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:18.352591 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.353123 2551 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.349516 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:18.354866 1928 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:18.162653 5114 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.356959 2580 pending_rounds.cc:85] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting all ops after (but not including) 7
I20250626 08:24:18.361187 2580 pending_rounds.cc:107] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250626 08:24:18.361500 2580 raft_consensus.cc:2887] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:18.301453 5141 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:18.352459 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.331691 4973 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:24:18.376523 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 70422.2us
I20250626 08:24:18.377362 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.351480 4832 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:18.351104 1812 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:18.349002 5084 raft_consensus.cc:2747] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:18.345947 5119 raft_consensus.cc:2802] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:18.356621 5076 raft_consensus.cc:2747] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.379871 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:18.380196 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.293071 5037 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.382047 4973 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:18.382558 4634 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:18.348743 4496 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:18.368956 1850 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:18.301180 5030 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:18.455891 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.456020 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.347079 4948 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:18.312731 5048 consensus_queue.cc:1035] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20250626 08:24:18.406957 1918 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:18.361369 4943 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.456797 1850 raft_consensus.cc:2391] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 10: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:18.472932 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.425309 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 48008.3us
I20250626 08:24:18.390978 1995 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:18.474623 1968 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.360392 5142 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.360848 5060 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:18.464847 1998 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:18.419206 1849 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:18.477810 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.507767 1850 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:18.397004 4832 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.511307 4832 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.508510 5060 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:18.512096 4832 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:18.512320 5060 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:18.512650 4832 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:18.514752 4832 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:18.479841 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 177375us
I20250626 08:24:18.476996 5048 consensus_queue.cc:1230] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250626 08:24:18.515928 1893 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.463034 5119 raft_consensus.cc:695] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:18.509195 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.517154 1893 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.518844 5119 consensus_queue.cc:237] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.511197 1918 raft_consensus.cc:1238] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 7. Ops: []
W20250626 08:24:18.477108 5121 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250626 08:24:18.520092 1893 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:18.520356 4832 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.455114 4634 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:18.395884 5133 raft_consensus.cc:2747] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:18.522485 4832 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.480535 5063 consensus_queue.cc:237] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.524103 4832 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.525697 1893 leader_election.cc:400] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:18.526325 1893 leader_election.cc:403] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:18.527071 4832 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:18.470863 5024 consensus_queue.cc:237] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.528842 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:18.530004 4832 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:18.530726 4832 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:18.461401 4496 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:18.531340 4832 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:18.469101 5157 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:18.533406 1893 leader_election.cc:400] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:18.534018 5157 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.534727 1893 leader_election.cc:403] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:18.537110 5157 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:18.539040 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 71624.1us
I20250626 08:24:18.522588 4634 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.554440 5132 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.454366 4973 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.378070 5029 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.350711 4486 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:18.551716 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 72683.8us
I20250626 08:24:18.460361 5141 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:18.469764 4948 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:18.633059 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1950) took 1387 ms. Trace:
I20250626 08:24:18.633494 1927 rpcz_store.cc:276] 0626 08:24:17.245568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:17.930188 (+684620us) service_pool.cc:225] Handling call
0626 08:24:17.930495 (+ 307us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:18.045808 (+115313us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:18.045838 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:18.054530 (+ 8692us) log.cc:844] Serialized 26 byte log entry
0626 08:24:18.244637 (+190107us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:18.244697 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.244792 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:18.632796 (+388004us) raft_consensus.cc:1683] finished
0626 08:24:18.632815 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.632994 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45355,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"thread_start_us":251215,"threads_started":2,"wal-append.queue_time_us":302621}
I20250626 08:24:18.417820 5113 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.149s
I20250626 08:24:18.531796 4496 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
W20250626 08:24:18.619014 2551 leader_election.cc:336] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.428s (SENT)
W20250626 08:24:18.552661 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.625345 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 150841us
I20250626 08:24:18.653211 1968 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.630328 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:18.562808 1893 connection.cc:232] RPC call timeout handler was delayed by -2.0232803940002668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20250626 08:24:18.639735 5029 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.610028 4928 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:18.643966 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 527) took 4015 ms. Trace:
W20250626 08:24:18.650179 4960 consensus_meta.cc:287] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.678s user 0.009s sys 0.004s
I20250626 08:24:18.666417 4928 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.668387 5044 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250626 08:24:18.668658 4928 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:18.629788 4979 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:18.669476 4979 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:18.670547 5147 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20250626 08:24:18.671638 4979 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.672963 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1949) took 1433 ms. Trace:
W20250626 08:24:18.650769 5151 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.276s user 0.000s sys 0.001s
W20250626 08:24:18.654326 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9935803940002188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
W20250626 08:24:18.686847 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9935803940002188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
W20250626 08:24:18.687331 1893 connection.cc:232] RPC call timeout handler was delayed by -1.9593803940001635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20250626 08:24:18.687703 1893 connection.cc:232] RPC call timeout handler was delayed by -1.859480394000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20250626 08:24:18.688098 1893 connection.cc:232] RPC call timeout handler was delayed by -1.8459803939999802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20250626 08:24:18.688468 1893 connection.cc:232] RPC call timeout handler was delayed by -1.8369803939999656s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20250626 08:24:18.688834 1893 connection.cc:232] RPC call timeout handler was delayed by -1.8369803939999656s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20250626 08:24:18.689216 1893 connection.cc:232] RPC call timeout handler was delayed by -1.6605803940001351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.213s for a response
W20250626 08:24:18.689579 1893 connection.cc:232] RPC call timeout handler was delayed by -1.5777803940000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20250626 08:24:18.660023 1843 rpcz_store.cc:276] 0626 08:24:14.628857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.628993 (+ 136us) service_pool.cc:225] Handling call
0626 08:24:14.685392 (+ 56399us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.121265 (+3435873us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.643489 (+522224us) spinlock_profiling.cc:243] Waited 3.43 s on lock 0x7b54000343d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:18.643863 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7882170112}
W20250626 08:24:18.691604 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3824803940001402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2439s for a response
I20250626 08:24:18.662245 1850 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:18.673463 1921 rpcz_store.cc:276] 0626 08:24:17.239822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:17.894533 (+654711us) service_pool.cc:225] Handling call
0626 08:24:17.894905 (+ 372us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:17.895274 (+ 369us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:17.895311 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:17.902261 (+ 6950us) log.cc:844] Serialized 26 byte log entry
0626 08:24:18.124571 (+222310us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:18.124771 (+ 200us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.125139 (+ 368us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:18.672706 (+547567us) raft_consensus.cc:1683] finished
0626 08:24:18.672730 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.672892 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":80692,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":82,"thread_start_us":157348,"threads_started":1,"wal-append.queue_time_us":412249}
I20250626 08:24:18.684708 1848 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:18.660449 1968 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.694530 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:18.695608 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.697690 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:18.692092 1893 connection.cc:232] RPC call timeout handler was delayed by -1.3824803940001402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2439s for a response
I20250626 08:24:18.699537 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:18.633155 5162 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:18.699988 4911 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.673230 5007 raft_consensus.cc:3053] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:24:18.706200 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 45822us
I20250626 08:24:18.697070 5075 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:18.725509 1843 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:18.723944 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.726274 1998 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:18.731784 1893 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.731863 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.733423 1893 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.733822 1998 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:18.739874 1968 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.631407 5038 raft_consensus.cc:3053] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:18.715783 5007 raft_consensus.cc:738] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:18.743860 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.720208 4979 consensus_queue.cc:1035] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20250626 08:24:18.741283 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:18.791455 5007 consensus_queue.cc:260] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.748560 5133 raft_consensus.cc:2747] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20250626 08:24:18.749531 5154 log.cc:927] Time spent T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20250626 08:24:18.728495 4911 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250626 08:24:18.752898 5047 raft_consensus.cc:3053] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:18.759750 1998 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:24:18.762491 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.773396 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.802484 2551 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:18.756311 5038 raft_consensus.cc:738] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:18.801421 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.755874 4662 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:18.739483 5158 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:18.819564 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.824122 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.825529 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.798964 5037 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.821008 4928 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:18.797369 5159 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
W20250626 08:24:18.836678 1816 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.012s sys 0.021s
W20250626 08:24:18.837070 1816 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.509s user 0.013s sys 0.021s
I20250626 08:24:18.806576 5030 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.837671 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.804630 5047 raft_consensus.cc:738] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
W20250626 08:24:18.837421 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 510064us
W20250626 08:24:18.829730 5160 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20250626 08:24:18.812270 4662 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:18.842692 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.871366 1998 raft_consensus.cc:2366] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
W20250626 08:24:18.867738 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.873312 4495 consensus_meta.cc:287] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.671s user 0.007s sys 0.003s
I20250626 08:24:18.875393 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:18.872591 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.830538 5114 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:18.833897 5158 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:18.836594 4928 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:18.889415 1816 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:18.887892 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.893185 4928 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:18.875828 4662 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:18.899601 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.905534 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:18.906163 1998 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:18.907793 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.909498 4960 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:18.910120 4960 raft_consensus.cc:695] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:18.893352 5037 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:18.910833 5037 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.910974 4960 consensus_queue.cc:237] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.911370 4832 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.912046 4840 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.912883 5037 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.913349 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2090) took 4275 ms. Trace:
I20250626 08:24:18.913856 1926 rpcz_store.cc:276] 0626 08:24:14.637580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.637738 (+ 158us) service_pool.cc:225] Handling call
0626 08:24:14.640712 (+ 2974us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.619972 (+1979260us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.913134 (+2293162us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54001f9354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:18.913293 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4046451456}
I20250626 08:24:18.913417 4832 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:18.920315 1995 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250626 08:24:18.894656 4486 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:18.906642 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 101749us
W20250626 08:24:18.927727 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 194396us
I20250626 08:24:18.928707 1893 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:18.929759 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:18.881181 4634 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.937775 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.936293 1893 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.876658 4495 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.940258 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:18.940563 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 200769us
I20250626 08:24:18.941989 1893 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:18.924965 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:18.950330 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:18.954425 1812 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items
W20250626 08:24:18.935808 5004 consensus_meta.cc:287] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.820s user 0.009s sys 0.001s
I20250626 08:24:18.955082 5004 raft_consensus.cc:2747] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.956014 5004 raft_consensus.cc:2802] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:18.956617 5004 raft_consensus.cc:695] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:18.954736 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1811)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1812)
I20250626 08:24:18.951045 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1951) took 1702 ms. Trace:
I20250626 08:24:18.958284 1928 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:18.958041 1923 rpcz_store.cc:276] 0626 08:24:17.248458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.270581 (+1022123us) service_pool.cc:225] Handling call
0626 08:24:18.271009 (+ 428us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:18.312441 (+ 41432us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:18.312477 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:18.335217 (+ 22740us) log.cc:844] Serialized 26 byte log entry
0626 08:24:18.792581 (+457364us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:18.792625 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.792771 (+ 146us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:18.950830 (+158059us) raft_consensus.cc:1683] finished
0626 08:24:18.950849 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.950990 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35187,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":19430,"thread_start_us":397077,"threads_started":1,"wal-append.queue_time_us":504366}
I20250626 08:24:18.961357 4634 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:18.962728 5116 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.889s user 0.003s sys 0.007s
I20250626 08:24:18.963156 5116 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.964670 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:18.965277 1923 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.966513 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:18.892232 5038 consensus_queue.cc:260] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.970458 5038 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:18.972100 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.972843 1923 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:18.973176 1997 raft_consensus.cc:2391] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:18.974627 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:18.944113 5158 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:18.975450 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1811)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1812)
I20250626 08:24:18.978792 1993 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20250626 08:24:18.978904 2000 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:18.979734 4495 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:18.981868 1968 leader_election.cc:336] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.190s (SENT)
W20250626 08:24:18.983590 4997 consensus_meta.cc:287] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.763s user 0.005s sys 0.005s
I20250626 08:24:18.984007 4997 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.984254 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1811)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1812)
I20250626 08:24:18.989231 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:18.956609 5098 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.976s user 0.010s sys 0.000s
I20250626 08:24:18.991955 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2093) took 4342 ms. Trace:
I20250626 08:24:18.991762 5098 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.992300 1924 rpcz_store.cc:276] 0626 08:24:14.649593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.718657 (+ 69064us) service_pool.cc:225] Handling call
0626 08:24:14.718956 (+ 299us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.635500 (+1916544us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.991722 (+2356222us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54002d05d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:18.991889 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3825552768}
I20250626 08:24:18.992760 1846 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:18.994971 5004 consensus_queue.cc:237] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:18.996829 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2227) took 2303 ms. Trace:
I20250626 08:24:18.915198 5047 consensus_queue.cc:260] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:18.999775 5047 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.002283 5106 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:19.002875 5106 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:18.994603 5098 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:18.975859 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.007421 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:19.008078 1997 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:19.009670 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.010083 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.010304 1924 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:19.010864 1923 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:18.997182 2002 rpcz_store.cc:276] 0626 08:24:16.693193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.694161 (+ 968us) service_pool.cc:225] Handling call
0626 08:24:16.694541 (+ 380us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:16.694807 (+ 266us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:17.060543 (+365736us) log.cc:844] Serialized 14 byte log entry
0626 08:24:18.996280 (+1935737us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b18001bb800. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b 00007f4eb8cc2b53 00007f4eb8cc306a
0626 08:24:18.996475 (+ 195us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:18.996527 (+ 52us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:18.996578 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.996651 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:18.996778 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32536,"prepare.queue_time_us":592921,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"spinlock_wait_cycles":3807488,"thread_start_us":575793,"threads_started":2,"wal-append.queue_time_us":804846,"wal-append.run_cpu_time_us":1369,"wal-append.run_wall_time_us":1070575}
I20250626 08:24:19.009892 1967 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:19.018599 1998 raft_consensus.cc:2408] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20250626 08:24:19.027247 1998 raft_consensus.cc:1238] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:18.987828 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1811)
I20250626 08:24:19.031488 1816 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:18.943375 4832 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:19.035722 1919 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:19.044230 1998 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:18.943157 1893 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:19.045658 1998 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
W20250626 08:24:19.052059 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 108988us
I20250626 08:24:19.054567 1893 leader_election.cc:400] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:19.054961 1893 leader_election.cc:403] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:19.056887 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.060307 1923 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:19.021860 5116 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.065217 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:19.033119 4643 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:19.033618 5098 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:19.075619 1967 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.084131 1997 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 10.
I20250626 08:24:19.088776 2002 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 9. Current term is 10. Ops: [7.6-9.7]
I20250626 08:24:19.090368 1816 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:19.056293 4634 raft_consensus.cc:2802] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:19.086397 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:19.068010 4832 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:19.029568 4928 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:19.127709 5147 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250626 08:24:19.125870 5116 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:19.074892 5158 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 755d83ca0913406b816e95415d502055 (127.0.171.195:38293) with higher term. Message: Invalid argument: T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:19.131533 4807 raft_consensus.cc:2802] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader election won for term 10
I20250626 08:24:19.086802 4486 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20250626 08:24:19.119176 5098 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.162714 2578 raft_consensus.cc:2366] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
I20250626 08:24:19.149938 4832 raft_consensus.cc:695] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.179819 1927 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.167520 4988 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.171058 5119 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.162520 4634 raft_consensus.cc:695] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:19.166313 5116 raft_consensus.cc:695] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:19.178773 4643 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:19.198483 2550 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.015s sys 0.043s
W20250626 08:24:19.200172 2550 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.605s user 0.015s sys 0.043s
W20250626 08:24:19.200524 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 605579us
I20250626 08:24:19.180439 4807 raft_consensus.cc:695] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:19.207342 2586 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250626 08:24:19.204010 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:19.208261 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:19.209177 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 154775us
W20250626 08:24:19.211088 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.205025 1812 leader_election.cc:400] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
W20250626 08:24:19.206177 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 222912us
I20250626 08:24:19.212386 1812 leader_election.cc:403] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250626 08:24:19.213722 1927 raft_consensus.cc:1273] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:19.215584 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:19.203366 5141 consensus_meta.cc:287] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.571s user 0.009s sys 0.002s
I20250626 08:24:19.229606 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.230477 2581 raft_consensus.cc:2366] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
W20250626 08:24:19.203822 5060 consensus_meta.cc:287] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.687s user 0.005s sys 0.006s
W20250626 08:24:19.217993 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.218302 5171 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 08:24:19.215777 5142 consensus_meta.cc:287] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.663s user 0.012s sys 0.000s
I20250626 08:24:19.220268 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.221133 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.221320 5176 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:19.216759 5004 consensus_queue.cc:1046] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20250626 08:24:19.222186 4928 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.207844 4643 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.224632 4807 consensus_queue.cc:237] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.209738 4662 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.226202 5098 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.226868 1924 raft_consensus.cc:2466] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9.
I20250626 08:24:19.226868 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:19.266011 1997 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:19.207036 5017 consensus_meta.cc:287] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.103s user 0.009s sys 0.004s
I20250626 08:24:19.266505 1967 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.266767 5017 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.268703 5017 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:19.233947 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2220) took 2960 ms (client timeout 2000 ms). Trace:
I20250626 08:24:19.220176 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:19.269336 1996 rpcz_store.cc:269] 0626 08:24:16.269157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.273047 (+ 3890us) service_pool.cc:225] Handling call
0626 08:24:19.228999 (+2955952us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b540039d854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.229101 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1029143808}
I20250626 08:24:19.269735 5133 raft_consensus.cc:2802] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:19.252283 5159 raft_consensus.cc:3053] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:19.270403 5133 raft_consensus.cc:695] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [term 9 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:19.240967 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.213374 4997 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.271497 5133 consensus_queue.cc:237] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.272078 4643 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.230907 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1811)
I20250626 08:24:19.273802 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.275015 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.242920 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:19.275723 5155 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.615s user 0.001s sys 0.000s
I20250626 08:24:19.242507 1923 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:19.276500 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.276973 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.244282 5142 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:19.277577 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.278048 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.196578 4988 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:19.231302 4948 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Time spent flushing consensus metadata: real 0.569s user 0.011s sys 0.000s
I20250626 08:24:19.280634 4988 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.245615 4495 consensus_queue.cc:1035] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:19.235414 4832 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.235121 4960 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:19.249858 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.251205 1921 raft_consensus.cc:1238] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:19.240270 5135 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20250626 08:24:19.259733 5188 raft_consensus.cc:3053] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:19.233299 5141 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:19.225761 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 453) took 1068 ms. Trace:
I20250626 08:24:19.289469 1993 rpcz_store.cc:276] 0626 08:24:18.157474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.161898 (+ 4424us) service_pool.cc:225] Handling call
0626 08:24:18.162327 (+ 429us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:18.979361 (+817034us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.225522 (+246161us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54001315d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:19.225695 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1871411456}
I20250626 08:24:19.250008 2578 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:19.236732 5060 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.270519 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:19.300057 5142 consensus_queue.cc:1046] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250626 08:24:19.301771 5142 raft_consensus.cc:3053] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Stepping down as leader of term 8
I20250626 08:24:19.302171 5142 raft_consensus.cc:738] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:19.303018 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.302963 5142 consensus_queue.cc:260] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.304926 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 553) took 1043 ms. Trace:
I20250626 08:24:19.305208 1846 rpcz_store.cc:276] 0626 08:24:18.261672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.274469 (+ 12797us) service_pool.cc:225] Handling call
0626 08:24:19.304796 (+1030327us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b5400107554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.304881 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1649837440}
I20250626 08:24:19.306156 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.307260 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.308007 1846 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:19.308708 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:19.310962 1925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39260 (request call id 1945) took 3001 ms (client timeout 1751 ms). Trace:
W20250626 08:24:19.311367 1925 rpcz_store.cc:269] 0626 08:24:16.309405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.343468 (+ 34063us) service_pool.cc:225] Handling call
0626 08:24:19.310775 (+2967307us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001e2154. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.310894 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2370648320}
I20250626 08:24:19.280299 4948 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.201157 5119 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.243028 4922 consensus_queue.cc:1046] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.005s
I20250626 08:24:19.273614 4938 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.242620 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.321594 2583 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:19.232872 5114 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20250626 08:24:19.287936 1845 raft_consensus.cc:3058] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.260929 4634 consensus_queue.cc:237] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.281788 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
I20250626 08:24:19.270430 5159 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.295863 2000 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250626 08:24:19.289186 5188 raft_consensus.cc:738] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.316193 1997 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:19.317206 1918 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:19.309661 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.318197 1890 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:19.336813 1846 raft_consensus.cc:2391] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:19.285691 2582 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:19.321630 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.323513 5164 raft_consensus.cc:3053] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:19.340409 5164 raft_consensus.cc:738] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:19.341190 1968 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:19.341383 5164 consensus_queue.cc:260] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:19.342516 4973 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.002s
I20250626 08:24:19.342751 2000 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:19.343057 4973 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.343451 5179 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250626 08:24:19.343658 4662 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:19.297221 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.328884 1812 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.338290 1923 raft_consensus.cc:1238] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 9. Current term is 10. Ops: []
I20250626 08:24:19.338465 2551 leader_election.cc:304] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:19.291857 5116 consensus_queue.cc:237] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.349771 1890 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.350090 1845 raft_consensus.cc:2466] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:19.337349 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1909) took 1036 ms. Trace:
W20250626 08:24:19.352694 5079 log.cc:927] Time spent T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.867s user 0.011s sys 0.016s
I20250626 08:24:19.353554 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.352434 5189 consensus_queue.cc:1046] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:19.364722 1921 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 8: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:19.355765 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 552) took 1094 ms. Trace:
I20250626 08:24:19.353359 5047 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:19.351487 1846 rpcz_store.cc:276] 0626 08:24:18.300570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.309444 (+1008874us) service_pool.cc:225] Handling call
0626 08:24:19.337292 (+ 27848us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.366047 1845 rpcz_store.cc:276] 0626 08:24:18.261408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.249590 (+988182us) service_pool.cc:225] Handling call
0626 08:24:19.355690 (+106100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.355136 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1931)
I20250626 08:24:19.340466 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:19.407287 1999 raft_consensus.cc:2391] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:19.408608 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
W20250626 08:24:19.415067 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:19.415908 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 55528.6us
I20250626 08:24:19.355191 5141 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250626 08:24:19.415992 5159 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.423846 5159 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:19.416101 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1937)
I20250626 08:24:19.365247 5164 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:19.427119 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1810)
I20250626 08:24:19.363976 5006 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.432610 5006 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.436264 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.436899 2000 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:19.438113 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:19.438710 4496 consensus_meta.cc:287] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.757s user 0.007s sys 0.004s
I20250626 08:24:19.439273 4496 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.440490 1998 raft_consensus.cc:1238] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:19.441068 5006 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.441514 4496 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:19.442485 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.442921 5119 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:19.443858 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.445254 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:19.446064 5006 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:19.446463 5006 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:19.446799 5006 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:19.451112 4973 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.366910 5004 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:19.432616 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1801)
I20250626 08:24:19.355551 5135 raft_consensus.cc:2802] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:19.351315 5098 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.102s
I20250626 08:24:19.462574 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555)
W20250626 08:24:19.470067 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 53319.5us
I20250626 08:24:19.471474 1890 leader_election.cc:400] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:19.471861 1890 leader_election.cc:403] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:19.453073 5141 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:19.477993 1812 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 555) took 1089 ms. Trace:
I20250626 08:24:19.487244 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:19.488258 2578 raft_consensus.cc:2391] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:19.488418 2000 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:19.491094 4832 raft_consensus.cc:2747] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:19.491943 4832 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 08:24:19.495419 4960 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733) with higher term. Message: Invalid argument: T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6.
I20250626 08:24:19.483086 1812 rpcz_store.cc:276] 0626 08:24:18.389110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.477891 (+1088781us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:19.505950 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:19.506764 1849 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1915) took 1187 ms (client timeout 1535 ms). Trace:
W20250626 08:24:19.507180 1849 rpcz_store.cc:269] 0626 08:24:18.319102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.323643 (+ 4541us) service_pool.cc:225] Handling call
0626 08:24:19.506581 (+1182938us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b5400211454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.506703 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":151076224}
I20250626 08:24:19.519934 1848 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:19.521584 1993 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
W20250626 08:24:19.549852 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.480675 5135 raft_consensus.cc:695] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:19.498654 5141 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:19.506330 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41946 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
I20250626 08:24:19.507216 5060 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.464259 5188 consensus_queue.cc:260] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.580763 5188 raft_consensus.cc:3058] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.582137 4948 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.591233 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1808) took 1038 ms. Trace:
W20250626 08:24:19.591564 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.591626 1843 rpcz_store.cc:276] 0626 08:24:18.552474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.694252 (+141778us) service_pool.cc:225] Handling call
0626 08:24:19.591086 (+896834us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b540032ebd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.591182 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69708672}
I20250626 08:24:19.592655 5006 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:19.595685 5097 log.cc:927] Time spent T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 2.263s user 0.005s sys 0.005s
I20250626 08:24:19.588506 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
W20250626 08:24:19.573213 5081 log.cc:927] Time spent T 3946d60648894026984085c3abee51ea P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250626 08:24:19.577080 5124 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
W20250626 08:24:19.603657 5082 consensus_queue.cc:1162] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" responder_term: 7 status { last_received { term: 5 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250626 08:24:19.606037 1999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 457) took 1226 ms. Trace:
I20250626 08:24:19.606448 1999 rpcz_store.cc:276] 0626 08:24:18.379500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.379642 (+ 142us) service_pool.cc:225] Handling call
0626 08:24:19.605756 (+1226114us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b5400271854. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.605943 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2236963200}
I20250626 08:24:19.608271 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:19.609335 1924 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:19.616510 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 528) took 4987 ms. Trace:
W20250626 08:24:19.620862 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 149573us
I20250626 08:24:19.617053 1842 rpcz_store.cc:276] 0626 08:24:14.629260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.586544 (+1957284us) service_pool.cc:225] Handling call
0626 08:24:16.586994 (+ 450us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:16.609706 (+ 22712us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:16.609750 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:16.638634 (+ 28884us) log.cc:844] Serialized 78 byte log entry
0626 08:24:17.011320 (+372686us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:18.053007 (+1041687us) log.cc:844] Serialized 14 byte log entry
0626 08:24:18.054685 (+ 1678us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:18.054814 (+ 129us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:19.595032 (+1540218us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:19.605576 (+ 10544us) raft_consensus.cc:1683] finished
0626 08:24:19.616214 (+ 10638us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:19.616437 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16521,"prepare.queue_time_us":576234,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":8440,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":18401,"spinlock_wait_cycles":27513088,"thread_start_us":1197853,"threads_started":2,"wal-append.queue_time_us":565801}
I20250626 08:24:19.621985 1890 leader_election.cc:304] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.623252 5188 raft_consensus.cc:2747] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:19.623800 1890 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:19.626672 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.601212 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
I20250626 08:24:19.629297 4662 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.616015 5065 raft_consensus.cc:2747] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250626 08:24:19.642829 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.643441 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:19.643893 1993 raft_consensus.cc:1273] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:19.644192 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.644932 1921 raft_consensus.cc:2374] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7. Re-sending same reply.
I20250626 08:24:19.647190 1921 raft_consensus.cc:1238] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20250626 08:24:19.646080 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1828)
W20250626 08:24:19.649259 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.650853 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:19.650166 1968 consensus_peers.cc:487] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:19.652562 2550 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:19.652567 5120 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.006s
W20250626 08:24:19.660020 1968 consensus_peers.cc:487] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250626 08:24:19.660162 1893 consensus_peers.cc:487] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250626 08:24:19.660892 1968 consensus_peers.cc:487] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60136 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:19.661175 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 515) took 5230 ms. Trace:
I20250626 08:24:19.662894 5006 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.661564 1841 rpcz_store.cc:276] 0626 08:24:14.430348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.430479 (+ 131us) service_pool.cc:225] Handling call
0626 08:24:14.465010 (+ 34531us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:14.524507 (+ 59497us) raft_consensus.cc:1532] Early marking committed up to index 6
0626 08:24:14.904477 (+379970us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b1800194380. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb6102bb5 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611d555 00007f4ebca62657 00007f4eb9089587 00007f4eb9098c9f 00007f4eb9098c18
0626 08:24:16.011678 (+1107201us) log.cc:844] Serialized 14 byte log entry
0626 08:24:16.372678 (+361000us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:16.382028 (+ 9350us) log.cc:844] Serialized 26 byte log entry
0626 08:24:16.382193 (+ 165us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:16.385723 (+ 3530us) log.cc:844] Serialized 14 byte log entry
0626 08:24:16.496033 (+110310us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:16.496119 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:17.807748 (+1311629us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b4800080358. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebdf36871 00007f4ebdf2da17 00007f4ebdf2d8aa 00007f4ebe02297f 00007f4ebe022647 00007f4ebe036c6c 00007f4ebe036c18 00007f4ebe036b98 00007f4ebe036b53 00007f4ebe035dfc 00007f4ebcfbc956
0626 08:24:19.244739 (+1436991us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b1800194380. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1
0626 08:24:19.600121 (+355382us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b1800194380. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:19.660933 (+ 60812us) raft_consensus.cc:1683] finished
0626 08:24:19.660954 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:19.661089 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":146032,"prepare.queue_time_us":1482930,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":2374,"raft.queue_time_us":199100,"raft.run_cpu_time_us":28199,"raft.run_wall_time_us":1436203,"spinlock_wait_cycles":37326208,"thread_start_us":1378068,"threads_started":2,"wal-append.queue_time_us":338079}
I20250626 08:24:19.639243 5114 consensus_queue.cc:1046] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250626 08:24:19.663848 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.666301 5065 raft_consensus.cc:2802] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:19.666872 5065 raft_consensus.cc:695] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:19.644644 4662 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.667845 5065 consensus_queue.cc:237] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.668802 4662 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.667583 1812 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:60106 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1828)
I20250626 08:24:19.691959 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1956) took 2106 ms. Trace:
I20250626 08:24:19.696827 1928 rpcz_store.cc:276] 0626 08:24:17.585458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.355879 (+770421us) service_pool.cc:225] Handling call
0626 08:24:18.374880 (+ 19001us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.959057 (+584177us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.678456 (+719399us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54001e1c54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:19.691899 (+ 13443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1341114240}
W20250626 08:24:19.697825 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.699782 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.691798 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
I20250626 08:24:19.691855 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.695282 1925 raft_consensus.cc:1238] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:19.695786 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1811) took 1048 ms. Trace:
W20250626 08:24:19.696290 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1908) took 1396 ms (client timeout 1830 ms). Trace:
I20250626 08:24:19.652114 5135 consensus_queue.cc:237] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.692752 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1797) took 1429 ms. Trace:
I20250626 08:24:19.702713 1921 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:19.703166 1848 rpcz_store.cc:276] 0626 08:24:18.263406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.699255 (+435849us) service_pool.cc:225] Handling call
0626 08:24:19.673109 (+973854us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54000357d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.692702 (+ 19593us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1824940928}
I20250626 08:24:19.703619 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:19.704468 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.706501 1841 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:19.702714 2002 rpcz_store.cc:276] 0626 08:24:18.647585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.030399 (+382814us) service_pool.cc:225] Handling call
0626 08:24:19.088539 (+ 58140us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:19.089218 (+ 679us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.684943 (+595725us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b54000794d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.695730 (+ 10787us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":132776832}
I20250626 08:24:19.707207 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1802) took 1163 ms. Trace:
I20250626 08:24:19.707528 1841 rpcz_store.cc:276] 0626 08:24:18.544046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.663582 (+1119536us) service_pool.cc:225] Handling call
0626 08:24:19.707153 (+ 43571us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:19.702831 1850 rpcz_store.cc:269] 0626 08:24:18.300315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.508953 (+208638us) service_pool.cc:225] Handling call
0626 08:24:19.684344 (+1175391us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54002ba2d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.696236 (+ 11892us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53988224}
I20250626 08:24:19.710026 2579 raft_consensus.cc:1273] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEARNER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:19.710887 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.711383 5201 consensus_queue.cc:1035] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:24:19.712010 1967 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:19.713667 1844 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 535) took 5061 ms (client timeout 2019 ms). Trace:
W20250626 08:24:19.714232 1844 rpcz_store.cc:269] 0626 08:24:14.630844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:14.964274 (+333430us) service_pool.cc:225] Handling call
0626 08:24:19.684593 (+4720319us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b54002bbbd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.692157 (+ 7564us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5677968640}
I20250626 08:24:19.704836 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41946 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
I20250626 08:24:19.724094 1848 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:19.724879 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1910) took 1424 ms. Trace:
I20250626 08:24:19.725271 1848 rpcz_store.cc:276] 0626 08:24:18.300837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.704259 (+1403422us) service_pool.cc:225] Handling call
0626 08:24:19.724800 (+ 20541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.719681 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41946 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
I20250626 08:24:19.735563 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:19.736392 1997 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:19.729391 1812 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.750022 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.750855 1848 raft_consensus.cc:2374] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7. Re-sending same reply.
I20250626 08:24:19.751470 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1807) took 1201 ms. Trace:
I20250626 08:24:19.751843 1848 rpcz_store.cc:276] 0626 08:24:18.550537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.749675 (+1199138us) service_pool.cc:225] Handling call
0626 08:24:19.751399 (+ 1724us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.752952 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.753835 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1814) took 1193 ms. Trace:
I20250626 08:24:19.757248 1848 rpcz_store.cc:276] 0626 08:24:18.560303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.752640 (+1192337us) service_pool.cc:225] Handling call
0626 08:24:19.753770 (+ 1130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:19.763477 1893 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.748397 1965 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.764529 1812 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:19.765789 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.767796 1848 raft_consensus.cc:2408] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250626 08:24:19.768513 1997 raft_consensus.cc:2466] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:19.712029 5114 raft_consensus.cc:3053] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:19.770617 5114 raft_consensus.cc:738] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:19.771842 5114 consensus_queue.cc:260] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.775193 1993 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 9. Current term is 10. Ops: []
I20250626 08:24:19.666321 5004 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:24:19.787496 2551 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:19.766798 1893 consensus_peers.cc:487] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:19.789187 1893 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:19.794572 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1938)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 565)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1831)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1832)
I20250626 08:24:19.746788 5133 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:19.801399 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 549) took 1540 ms. Trace:
I20250626 08:24:19.763451 5142 raft_consensus.cc:3058] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:19.811004 1848 rpcz_store.cc:276] 0626 08:24:18.260533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.765401 (+1504868us) service_pool.cc:225] Handling call
0626 08:24:19.801261 (+ 35860us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:19.787245 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.801678 1812 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1938)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 565)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1831)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1832)
I20250626 08:24:19.816593 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.817317 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1798) took 1553 ms. Trace:
I20250626 08:24:19.786990 4960 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:19.817642 1848 rpcz_store.cc:276] 0626 08:24:18.263699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.815361 (+1551662us) service_pool.cc:225] Handling call
0626 08:24:19.817262 (+ 1901us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.777401 4988 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:19.774353 4989 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:19.821148 1893 consensus_peers.cc:487] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
I20250626 08:24:19.793900 4956 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.738648 4973 consensus_queue.cc:1046] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20250626 08:24:19.802145 4938 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:19.829788 1733 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:60106 dropped due to backpressure: service queue is full with 50 items [suppressed 44 similar messages]
I20250626 08:24:19.810962 5133 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:19.746212 5159 consensus_queue.cc:1046] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20250626 08:24:19.832844 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.802368 5188 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:19.830035 4956 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.777855 5120 consensus_queue.cc:1046] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20250626 08:24:19.847096 1848 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:19.856540 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.810037 4662 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:19.848618 5188 raft_consensus.cc:695] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.823093 4960 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:19.812418 5123 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.858530 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.821460 5176 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:19.744764 5177 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
W20250626 08:24:19.846397 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.861479 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1803) took 1316 ms. Trace:
I20250626 08:24:19.850157 5133 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:19.867851 1848 rpcz_store.cc:276] 0626 08:24:18.544822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.856300 (+1311478us) service_pool.cc:225] Handling call
0626 08:24:19.861417 (+ 5117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.871275 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.871886 1848 raft_consensus.cc:2366] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:19.874197 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1809) took 1320 ms. Trace:
I20250626 08:24:19.874584 1848 rpcz_store.cc:276] 0626 08:24:18.553338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.871043 (+1317705us) service_pool.cc:225] Handling call
0626 08:24:19.874103 (+ 3060us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.861251 5114 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.875540 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.826289 5200 consensus_queue.cc:1046] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20250626 08:24:19.834854 4989 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.866912 4960 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:19.863909 5030 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:19.889093 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1816) took 1326 ms. Trace:
I20250626 08:24:19.868319 5004 raft_consensus.cc:3053] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Stepping down as leader of term 9
I20250626 08:24:19.905280 5004 raft_consensus.cc:738] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.906453 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.907167 1928 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:19.913085 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.914234 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:19.915731 5162 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:19.916095 5162 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:19.916196 5063 raft_consensus.cc:3053] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:19.875701 5179 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:19.916831 5063 raft_consensus.cc:738] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:19.916972 5179 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:19.917440 5179 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:19.863974 4662 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:19.887313 5098 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:19.854517 4938 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.916455 5162 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:19.871621 5123 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.895326 4956 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.925765 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1895) took 5196 ms. Trace:
W20250626 08:24:19.927939 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.940966 1847 rpcz_store.cc:276] 0626 08:24:14.729205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.634023 (+1904818us) service_pool.cc:225] Handling call
0626 08:24:18.108219 (+1474196us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.120820 (+ 12601us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.925538 (+1804718us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540038c050. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.925707 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3389519616}
I20250626 08:24:19.941998 1848 rpcz_store.cc:276] 0626 08:24:18.562227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.875266 (+1313039us) service_pool.cc:225] Handling call
0626 08:24:19.889014 (+ 13748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.943750 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.944469 1848 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:19.937776 1843 raft_consensus.cc:2391] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:19.945509 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:19.923940 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1809) took 1293 ms. Trace:
I20250626 08:24:19.946635 1995 rpcz_store.cc:276] 0626 08:24:18.630565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.631205 (+ 640us) service_pool.cc:225] Handling call
0626 08:24:18.919963 (+288758us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:18.920870 (+ 907us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.923759 (+1002889us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003bef50. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:19.923883 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":663286528}
W20250626 08:24:19.949110 5203 log.cc:927] Time spent T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250626 08:24:19.949188 4662 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:19.951193 4938 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.952127 5094 consensus_queue.cc:1035] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:19.952838 5094 consensus_queue.cc:1230] T 46151fddbbf14be68feb086aa1fee4d9 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250626 08:24:19.953516 5123 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:19.958130 1928 raft_consensus.cc:1273] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250626 08:24:19.947391 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.959915 1848 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:19.960394 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1920) took 1639 ms. Trace:
I20250626 08:24:19.960672 1848 rpcz_store.cc:276] 0626 08:24:18.331139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.947147 (+1616008us) service_pool.cc:225] Handling call
0626 08:24:19.960343 (+ 13196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.961246 1921 pending_rounds.cc:85] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 6
I20250626 08:24:19.961715 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:19.952152 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.887053 5119 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:19.919827 5186 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.700s user 0.000s sys 0.002s
I20250626 08:24:19.962929 2579 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:19.964123 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.966020 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:19.964737 1843 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1911) took 1663 ms (client timeout 2146 ms). Trace:
I20250626 08:24:19.961624 1921 pending_rounds.cc:107] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250626 08:24:19.975033 1812 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:19.975324 1847 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:19.939881 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.978044 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1916) took 1658 ms. Trace:
I20250626 08:24:19.978590 1847 rpcz_store.cc:276] 0626 08:24:18.319445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.942289 (+1622844us) service_pool.cc:225] Handling call
0626 08:24:19.977982 (+ 35693us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.976503 1848 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:19.965484 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:19.980130 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 557) took 1282 ms. Trace:
I20250626 08:24:19.980502 2580 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
W20250626 08:24:19.976220 1843 rpcz_store.cc:269] 0626 08:24:18.301084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.912724 (+1611640us) service_pool.cc:225] Handling call
0626 08:24:19.964672 (+ 51948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.975373 1921 raft_consensus.cc:2887] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:19.988503 1812 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:19.965157 2581 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:19.981218 5029 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:19.993880 5029 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:19.994303 5029 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:19.995290 4989 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:19.984544 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:19.905336 5200 raft_consensus.cc:3053] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Stepping down as leader of term 9
I20250626 08:24:19.997320 5200 raft_consensus.cc:738] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:19.980800 1848 rpcz_store.cc:276] 0626 08:24:18.697799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.963913 (+1266114us) service_pool.cc:225] Handling call
0626 08:24:19.980059 (+ 16146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:19.977854 5188 consensus_queue.cc:237] T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.000115 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:19.963996 4956 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:20.001132 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.001542 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.008937 1848 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:20.000970 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:20.019294 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.993440 5123 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:20.008558 4956 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:19.985821 4973 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.026309 5123 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:20.026615 4956 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.026757 5123 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:20.026880 5201 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:20.002730 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.028606 1843 raft_consensus.cc:2374] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5. Re-sending same reply.
I20250626 08:24:20.017784 4832 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:20.029253 4832 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.026770 4973 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.031255 4832 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.032639 4973 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:20.032395 5162 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:19.922288 5030 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.005005 5098 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.034667 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:20.029531 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.004619 1847 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:20.035467 1924 raft_consensus.cc:2408] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:20.035663 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 554) took 1773 ms. Trace:
I20250626 08:24:19.959657 4495 consensus_queue.cc:1035] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
W20250626 08:24:19.965304 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 216998us
I20250626 08:24:20.036170 1847 rpcz_store.cc:276] 0626 08:24:18.261969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.979235 (+1717266us) service_pool.cc:225] Handling call
0626 08:24:20.035587 (+ 56352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.038101 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1924) took 1089 ms. Trace:
I20250626 08:24:20.040248 1843 rpcz_store.cc:276] 0626 08:24:18.949010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.029263 (+1080253us) service_pool.cc:225] Handling call
0626 08:24:20.038042 (+ 8779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.041939 2551 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:20.032790 5094 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:20.042619 1968 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:20.043298 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.044595 1843 raft_consensus.cc:2374] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7. Re-sending same reply.
W20250626 08:24:20.057124 1733 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 264000us
I20250626 08:24:20.076669 1924 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:20.080607 2551 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:19.906483 5004 consensus_queue.cc:260] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.011420 4662 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:19.978147 5119 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.085439 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:19.917802 5063 consensus_queue.cc:260] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.090211 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:19.985992 5024 consensus_queue.cc:1035] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250626 08:24:20.088935 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.064977 4948 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.100013 2586 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:20.093760 1843 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:20.002218 5141 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.097489 5183 log.cc:927] Time spent T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.875s user 0.003s sys 0.010s
I20250626 08:24:20.098166 5004 raft_consensus.cc:3058] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:20.073900 5094 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:20.079699 5046 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:20.074163 4973 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:20.104725 1968 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:20.106240 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:20.082751 5176 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:20.098224 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.113036 1848 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 5.
I20250626 08:24:20.095599 4918 raft_consensus.cc:2802] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:20.115954 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.132578 1843 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:20.114904 5094 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.129348 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.130177 1924 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:20.115130 5030 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:20.141074 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.161546 1848 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:20.074437 5200 consensus_queue.cc:260] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.172353 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1914) took 1853 ms. Trace:
I20250626 08:24:20.173022 1848 rpcz_store.cc:276] 0626 08:24:18.318697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.129080 (+1810383us) service_pool.cc:225] Handling call
0626 08:24:20.172275 (+ 43195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.174721 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.176059 1848 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:20.176923 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 558) took 1419 ms. Trace:
I20250626 08:24:20.177381 1848 rpcz_store.cc:276] 0626 08:24:18.757898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.174405 (+1416507us) service_pool.cc:225] Handling call
0626 08:24:20.176844 (+ 2439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.178838 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:20.180991 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.184332 1848 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 8: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:20.185819 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.190474 1848 raft_consensus.cc:2391] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:20.194057 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:20.204528 5207 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
W20250626 08:24:20.204869 5111 log.cc:927] Time spent T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 2.000s user 0.010s sys 0.003s
I20250626 08:24:20.205267 5214 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:20.205471 5141 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.206822 5049 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.207422 4960 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.210518 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:20.210657 5133 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.211365 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.212095 1846 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:20.115232 5046 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:20.104863 4948 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.215380 1968 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:20.213784 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:20.229107 1846 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1913) took 1927 ms (client timeout 2019 ms). Trace:
W20250626 08:24:20.229526 1846 rpcz_store.cc:269] 0626 08:24:18.301604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.211098 (+1909494us) service_pool.cc:225] Handling call
0626 08:24:20.229046 (+ 17948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.230433 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.230968 1846 raft_consensus.cc:3058] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:20.232007 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.134519 5162 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.233832 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:20.238101 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.238148 1848 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:20.115037 5208 consensus_queue.cc:1046] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20250626 08:24:20.159436 4918 raft_consensus.cc:695] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:20.163971 5159 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.253090 1847 raft_consensus.cc:2366] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:20.246272 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.247555 5133 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.252017 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2106) took 3609 ms. Trace:
I20250626 08:24:20.237062 5046 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:20.257061 5179 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.268205 5179 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.252163 5119 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.270663 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 445) took 3557 ms. Trace:
I20250626 08:24:20.271122 1994 rpcz_store.cc:276] 0626 08:24:16.713239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.743368 (+ 30129us) service_pool.cc:225] Handling call
0626 08:24:16.770638 (+ 27270us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:17.953666 (+1183028us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.270433 (+2316767us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540018e554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:20.270604 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2666132736}
I20250626 08:24:20.273406 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.235252 5060 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:20.255811 1928 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:20.247303 5135 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:20.234369 5030 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.256613 5159 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.131677 5029 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.257393 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.277721 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.257050 1920 rpcz_store.cc:276] 0626 08:24:16.642819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:16.872051 (+229232us) service_pool.cc:225] Handling call
0626 08:24:16.896534 (+ 24483us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:16.900718 (+ 4184us) raft_consensus.cc:1532] Early marking committed up to index 3
0626 08:24:16.933130 (+ 32412us) log.cc:844] Serialized 14 byte log entry
0626 08:24:17.295025 (+361895us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:17.297026 (+ 2001us) log.cc:844] Serialized 26 byte log entry
0626 08:24:17.297674 (+ 648us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:17.297825 (+ 151us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:17.298129 (+ 304us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:20.204580 (+2906451us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2400276d88. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb5f77bf4 00007f4eb5f75cfb 00007f4eb6004ab9 00007f4eb6004745 00007f4eb6003d8d 00007f4eb8ca6f9e 00007f4eb8cbe6cc 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051
0626 08:24:20.250737 (+ 46157us) raft_consensus.cc:1683] finished
0626 08:24:20.251810 (+ 1073us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:20.251956 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1200702,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":4116,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":8544,"spinlock_wait_cycles":24587008,"thread_start_us":241520,"threads_started":1,"wal-append.queue_time_us":1253383}
I20250626 08:24:20.282433 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.282968 2579 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.283682 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1804) took 1736 ms. Trace:
I20250626 08:24:20.284075 1848 rpcz_store.cc:276] 0626 08:24:18.547305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.277472 (+1730167us) service_pool.cc:225] Handling call
0626 08:24:20.283627 (+ 6155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.286345 2000 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250626 08:24:20.251228 5098 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:20.287881 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.289242 2001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 459) took 1658 ms. Trace:
I20250626 08:24:20.289608 2001 rpcz_store.cc:276] 0626 08:24:18.630698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.630891 (+ 193us) service_pool.cc:225] Handling call
0626 08:24:18.668010 (+ 37119us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:18.673290 (+ 5280us) raft_consensus.cc:1532] Early marking committed up to index 8
0626 08:24:18.673542 (+ 252us) log.cc:844] Serialized 14 byte log entry
0626 08:24:18.673723 (+ 181us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:18.673769 (+ 46us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:18.673807 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.289032 (+1615225us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b540039ce54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:20.289076 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:20.289192 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":910615,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":371,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":18125,"spinlock_wait_cycles":11538304}
I20250626 08:24:20.293378 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.259948 4979 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.248467 5123 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.299237 1924 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:20.261538 5208 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250626 08:24:20.281898 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.279433 5060 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:20.261436 4960 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.304942 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 1955) took 3019 ms. Trace:
W20250626 08:24:20.312139 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 96853.3us
I20250626 08:24:20.311866 1919 rpcz_store.cc:276] 0626 08:24:17.285046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.323073 (+1038027us) service_pool.cc:225] Handling call
0626 08:24:18.323403 (+ 330us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:19.036171 (+712768us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.304622 (+1268451us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b5400312ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:20.304864 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1637692800}
I20250626 08:24:20.311043 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:20.315716 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.316450 1968 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:20.320047 1848 raft_consensus.cc:2366] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 8. Current term is 9.
I20250626 08:24:20.321470 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.286898 5135 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.312395 5060 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:20.289819 5030 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.328799 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.331403 2579 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:20.332230 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:20.333940 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:20.259786 4918 consensus_queue.cc:237] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.334576 2001 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 10.
I20250626 08:24:20.335831 1812 leader_election.cc:304] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:20.333954 5208 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
W20250626 08:24:20.356860 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.353210 5146 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:20.399704 1843 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:20.396764 5106 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:20.395676 5211 log.cc:927] Time spent T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20250626 08:24:20.341725 4960 raft_consensus.cc:2802] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:20.407536 4960 raft_consensus.cc:695] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:20.400434 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1818) took 1119 ms. Trace:
I20250626 08:24:20.408229 1843 rpcz_store.cc:276] 0626 08:24:19.281394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.115679 (+834285us) service_pool.cc:225] Handling call
0626 08:24:20.400328 (+284649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.408488 4960 consensus_queue.cc:237] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.409485 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.410018 5159 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.410233 1843 raft_consensus.cc:2374] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7. Re-sending same reply.
I20250626 08:24:20.410426 5075 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:20.410952 5075 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:20.411065 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.411586 4956 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.414283 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:20.343263 5187 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 1.008s user 0.000s sys 0.003s
W20250626 08:24:20.343628 5210 log.cc:927] Time spent T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20250626 08:24:20.422209 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.347406 5201 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.353468 4916 raft_consensus.cc:3053] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:20.427992 2002 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:20.351116 5218 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:20.466140 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.466996 2579 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:20.420469 5116 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.026s sys 0.034s
I20250626 08:24:20.420990 5208 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:20.423010 4948 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.473955 1816 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:20.454625 4916 raft_consensus.cc:738] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:20.454200 4956 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.492827 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:20.472198 5029 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:20.496903 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 177645us
W20250626 08:24:20.483235 5116 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s user 0.029s sys 0.035s
I20250626 08:24:20.418727 5146 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.463487 5055 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20250626 08:24:20.485705 5188 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.506351 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:20.507195 2579 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:20.510200 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.511372 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:20.512418 2579 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:20.461478 5201 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.516839 1812 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:20.517035 2587 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250626 08:24:20.495287 5123 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.519404 1925 raft_consensus.cc:3053] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:20.499152 5098 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250626 08:24:20.513959 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 55242us
I20250626 08:24:20.519833 1925 raft_consensus.cc:738] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:20.507535 5119 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250626 08:24:20.479028 5218 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.522140 2002 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250626 08:24:20.623062 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.559233 5214 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250626 08:24:20.685705 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.520345 4989 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:20.523514 5119 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:20.514007 5002 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:20.570529 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:20.522046 1925 consensus_queue.cc:260] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.571645 5222 log.cc:927] Time spent T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20250626 08:24:20.513674 5188 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.520604 4916 consensus_queue.cc:260] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.578862 4662 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
W20250626 08:24:20.564455 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:20.572180 5221 log.cc:927] Time spent T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20250626 08:24:20.786481 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.828756 2579 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 9.
W20250626 08:24:20.829134 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:20.829941 2550 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:20.786322 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 263654us
W20250626 08:24:20.790911 1846 consensus_meta.cc:287] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.550s user 0.003s sys 0.014s
I20250626 08:24:20.794898 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 471) took 1061 ms. Trace:
I20250626 08:24:20.522974 5208 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.799391 5046 consensus_meta.cc:287] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.538s user 0.008s sys 0.010s
I20250626 08:24:20.822470 1925 raft_consensus.cc:3058] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:20.802632 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:20.822875 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.823055 1919 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:24:20.794822 5049 consensus_meta.cc:287] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.581s user 0.005s sys 0.014s
W20250626 08:24:20.795812 5226 log.cc:927] Time spent T 2717a332547d4ee8912bdda5311cffd8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.278s user 0.011s sys 0.000s
W20250626 08:24:20.796381 5225 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.285s user 0.000s sys 0.009s
I20250626 08:24:20.830173 1812 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:20.830513 1846 raft_consensus.cc:2466] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:20.795665 5036 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.830606 1996 rpcz_store.cc:276] 0626 08:24:19.733261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.734311 (+ 1050us) service_pool.cc:225] Handling call
0626 08:24:19.734787 (+ 476us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:19.747142 (+ 12355us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:19.747172 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:19.747801 (+ 629us) log.cc:844] Serialized 78 byte log entry
0626 08:24:19.983244 (+235443us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:19.990962 (+ 7718us) log.cc:844] Serialized 14 byte log entry
0626 08:24:19.992092 (+ 1130us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:19.992189 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:20.794671 (+802482us) raft_consensus.cc:1683] finished
0626 08:24:20.794691 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:20.794830 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18190,"prepare.queue_time_us":7433,"raft.queue_time_us":13815,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":33506,"spinlock_wait_cycles":16562560,"thread_start_us":169508,"threads_started":1,"wal-append.queue_time_us":292882}
I20250626 08:24:20.834939 2550 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:20.794274 4945 consensus_queue.cc:1046] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20250626 08:24:20.825666 4948 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.824434 5119 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Advancing to term 11
I20250626 08:24:20.839362 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1929) took 1852 ms. Trace:
I20250626 08:24:20.848235 1846 rpcz_store.cc:276] 0626 08:24:18.987439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.230188 (+1242749us) service_pool.cc:225] Handling call
0626 08:24:20.839315 (+609127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.849020 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.849524 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1938) took 1084 ms. Trace:
I20250626 08:24:20.849711 1846 rpcz_store.cc:276] 0626 08:24:19.765490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.848779 (+1083289us) service_pool.cc:225] Handling call
0626 08:24:20.849485 (+ 706us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.832746 4662 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:20.828850 5135 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.837360 5165 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.842664 5036 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.832506 4989 raft_consensus.cc:695] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:20.846211 5146 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.850378 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.858096 1846 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:20.841235 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.848031 5029 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:20.857110 1925 raft_consensus.cc:1273] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250626 08:24:20.860898 1925 pending_rounds.cc:85] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 7
I20250626 08:24:20.850529 5049 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:20.863683 5188 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:20.863863 5094 consensus_meta.cc:287] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.628s user 0.010s sys 0.006s
W20250626 08:24:20.864351 4979 consensus_meta.cc:287] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.559s user 0.006s sys 0.009s
I20250626 08:24:20.864318 5094 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.864833 4979 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:20.865455 2000 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
W20250626 08:24:20.866160 5060 consensus_meta.cc:287] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.526s user 0.005s sys 0.014s
I20250626 08:24:20.866583 5060 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.867126 5029 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:20.868108 5029 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.868795 5060 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:20.852667 4948 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:20.871587 4948 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.861625 1925 pending_rounds.cc:107] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250626 08:24:20.835882 5046 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.867128 5094 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:20.858328 5165 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.873857 1925 raft_consensus.cc:2887] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:20.874307 1846 raft_consensus.cc:2466] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
W20250626 08:24:20.874774 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:20.875461 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 562) took 1171 ms. Trace:
I20250626 08:24:20.875792 1846 rpcz_store.cc:276] 0626 08:24:19.704291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.850169 (+1145878us) service_pool.cc:225] Handling call
0626 08:24:20.875407 (+ 25238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.876837 2551 leader_election.cc:304] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:20.877187 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.877815 1846 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:20.878068 2587 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:20.857935 5201 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.881680 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:20.882347 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:20.884685 2585 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
W20250626 08:24:20.881048 4997 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20250626 08:24:20.892454 1846 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:20.893774 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.894717 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1824) took 1417 ms. Trace:
I20250626 08:24:20.895123 1846 rpcz_store.cc:276] 0626 08:24:19.477493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.893476 (+1415983us) service_pool.cc:225] Handling call
0626 08:24:20.894653 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.880954 5094 raft_consensus.cc:2802] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:20.875169 5049 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:20.905753 1841 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1800) took 2363 ms (client timeout 1957 ms). Trace:
W20250626 08:24:20.906368 1841 rpcz_store.cc:269] 0626 08:24:18.542422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.905538 (+2363116us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.905682 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:20.906688 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.907262 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.907421 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1918) took 2587 ms. Trace:
I20250626 08:24:20.907799 1846 rpcz_store.cc:276] 0626 08:24:18.320106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.906411 (+2586305us) service_pool.cc:225] Handling call
0626 08:24:20.907366 (+ 955us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.908772 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.909528 1846 raft_consensus.cc:2374] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6. Re-sending same reply.
I20250626 08:24:20.910190 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1925) took 1958 ms. Trace:
I20250626 08:24:20.910596 1846 rpcz_store.cc:276] 0626 08:24:18.952166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.908467 (+1956301us) service_pool.cc:225] Handling call
0626 08:24:20.910128 (+ 1661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.911741 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.912727 1846 raft_consensus.cc:2391] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:20.915104 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1940) took 1149 ms. Trace:
I20250626 08:24:20.915800 1846 rpcz_store.cc:276] 0626 08:24:19.766078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.911419 (+1145341us) service_pool.cc:225] Handling call
0626 08:24:20.915031 (+ 3612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.917487 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:20.918346 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.919090 1846 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.922466 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.927060 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.928884 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:20.884322 5157 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250626 08:24:20.932448 1918 raft_consensus.cc:1238] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:20.913669 5049 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:20.934392 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.890897 5201 consensus_queue.cc:1046] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20250626 08:24:20.939858 5201 raft_consensus.cc:3053] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:20.940234 5201 raft_consensus.cc:738] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:20.906172 5046 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:20.883311 4948 consensus_queue.cc:1046] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250626 08:24:20.941035 4948 consensus_queue.cc:1230] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer be56082cacb54e9aa8f1a9ee91e70d76 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250626 08:24:20.941588 5030 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:20.941004 5201 consensus_queue.cc:260] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.925390 5142 consensus_meta.cc:287] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.101s user 0.022s sys 0.012s
W20250626 08:24:20.932186 5114 consensus_meta.cc:287] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 1.035s user 0.008s sys 0.012s
I20250626 08:24:20.943069 5201 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.943508 5036 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:20.938918 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 551) took 2677 ms. Trace:
I20250626 08:24:20.945828 2581 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20250626 08:24:20.950847 2581 pending_rounds.cc:85] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting all ops after (but not including) 8
I20250626 08:24:20.950666 1841 rpcz_store.cc:276] 0626 08:24:18.261138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.931226 (+2670088us) service_pool.cc:225] Handling call
0626 08:24:20.938832 (+ 7606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.951242 2581 pending_rounds.cc:107] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20250626 08:24:20.951508 2581 raft_consensus.cc:2887] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250626 08:24:20.947348 1844 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1806) took 2397 ms (client timeout 2061 ms). Trace:
W20250626 08:24:20.952201 1844 rpcz_store.cc:269] 0626 08:24:18.549968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.947170 (+2397202us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.947283 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250626 08:24:20.946813 1849 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1796) took 2683 ms (client timeout 2130 ms). Trace:
W20250626 08:24:20.952877 1849 rpcz_store.cc:269] 0626 08:24:18.263133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.946597 (+2683464us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.946721 (+ 124us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:20.955823 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.956480 1844 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:20.957703 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:20.958041 1846 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:20.959079 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.959798 1846 raft_consensus.cc:2391] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:20.960747 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:20.963527 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.963667 4832 raft_consensus.cc:3053] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:20.964120 4832 raft_consensus.cc:738] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:24:20.964265 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1799) took 2424 ms (client timeout 2035 ms). Trace:
W20250626 08:24:20.964798 1850 rpcz_store.cc:269] 0626 08:24:18.540110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.964045 (+2423935us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.964188 (+ 143us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:20.965508 4916 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:20.965075 4832 consensus_queue.cc:260] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.966504 1850 raft_consensus.cc:1238] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: [5.7-6.8]
I20250626 08:24:20.966519 4832 raft_consensus.cc:3058] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
W20250626 08:24:20.970300 1842 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41946 (request call id 559) took 2148 ms (client timeout 1635 ms). Trace:
W20250626 08:24:20.970703 1842 rpcz_store.cc:269] 0626 08:24:18.822214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.970122 (+2147908us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.970236 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:20.971784 1842 raft_consensus.cc:1238] T fec12ab89e754fea8f2b9e353bba3dee P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:20.972354 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1795) took 2709 ms. Trace:
I20250626 08:24:20.972649 1842 rpcz_store.cc:276] 0626 08:24:18.262856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.971314 (+2708458us) service_pool.cc:225] Handling call
0626 08:24:20.971596 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.972107 (+ 511us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.972277 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.974471 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1898) took 2710 ms. Trace:
I20250626 08:24:20.974795 1842 rpcz_store.cc:276] 0626 08:24:18.264275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.973392 (+2709117us) service_pool.cc:225] Handling call
0626 08:24:20.973656 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.974150 (+ 494us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:20.974182 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:20.974233 (+ 51us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:20.974263 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.974323 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:20.974423 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.976264 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1901) took 2711 ms. Trace:
I20250626 08:24:20.976634 1842 rpcz_store.cc:276] 0626 08:24:18.265125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.975564 (+2710439us) service_pool.cc:225] Handling call
0626 08:24:20.975824 (+ 260us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.975983 (+ 159us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:20.976004 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:20.976039 (+ 35us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:20.976069 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.976117 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:20.976214 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.981390 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 598) took 1729 ms. Trace:
I20250626 08:24:20.982529 1842 raft_consensus.cc:1273] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:20.984000 1923 rpcz_store.cc:276] 0626 08:24:19.251992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.279684 (+ 27692us) service_pool.cc:225] Handling call
0626 08:24:19.279947 (+ 263us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:19.348986 (+ 69039us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.981092 (+1632106us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b54002dfad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:20.981313 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133760384}
I20250626 08:24:20.985332 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 550) took 2724 ms. Trace:
I20250626 08:24:20.985749 1850 rpcz_store.cc:276] 0626 08:24:18.260859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.965744 (+2704885us) service_pool.cc:225] Handling call
0626 08:24:20.966196 (+ 452us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:20.966984 (+ 788us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.985278 (+ 18294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.986176 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.987978 1844 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:20.988581 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:20.989213 1849 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:20.920406 4989 consensus_queue.cc:237] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:20.987188 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 478881us
W20250626 08:24:20.990478 1845 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1813) took 2431 ms (client timeout 1758 ms). Trace:
W20250626 08:24:20.990883 1845 rpcz_store.cc:269] 0626 08:24:18.558807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.990285 (+2431478us) service_pool.cc:212] Skipping call since client already timed out
0626 08:24:20.990417 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250626 08:24:20.991887 1845 raft_consensus.cc:1238] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:20.991897 1850 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:20.992513 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1904) took 2726 ms. Trace:
I20250626 08:24:20.992772 1845 rpcz_store.cc:276] 0626 08:24:18.266028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.991540 (+2725512us) service_pool.cc:225] Handling call
0626 08:24:20.991768 (+ 228us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.992305 (+ 537us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.992467 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.993526 1845 raft_consensus.cc:1238] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:20.994346 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1903) took 2728 ms. Trace:
I20250626 08:24:20.994781 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1905) took 2728 ms. Trace:
I20250626 08:24:20.994710 1850 rpcz_store.cc:276] 0626 08:24:18.265684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.991468 (+2725784us) service_pool.cc:225] Handling call
0626 08:24:20.991744 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.992467 (+ 723us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.994291 (+ 1824us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.995414 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:20.995244 1845 rpcz_store.cc:276] 0626 08:24:18.266321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.993223 (+2726902us) service_pool.cc:225] Handling call
0626 08:24:20.993421 (+ 198us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.993719 (+ 298us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.994734 (+ 1015us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.996243 1850 raft_consensus.cc:1273] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:20.996794 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1906) took 2730 ms. Trace:
I20250626 08:24:20.996910 1845 raft_consensus.cc:1238] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20250626 08:24:20.997133 1850 rpcz_store.cc:276] 0626 08:24:18.266590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.995525 (+2728935us) service_pool.cc:225] Handling call
0626 08:24:20.995782 (+ 257us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.996581 (+ 799us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.996750 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.997607 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1825) took 1409 ms. Trace:
I20250626 08:24:20.998101 1845 rpcz_store.cc:276] 0626 08:24:19.587848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.996337 (+1408489us) service_pool.cc:225] Handling call
0626 08:24:20.996676 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:20.997393 (+ 717us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.997566 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:20.998998 1845 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:20.999403 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1831) took 1205 ms. Trace:
I20250626 08:24:20.999657 1845 rpcz_store.cc:276] 0626 08:24:19.793678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.998710 (+1205032us) service_pool.cc:225] Handling call
0626 08:24:20.998894 (+ 184us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.999262 (+ 368us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.999371 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:21.002534 1844 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:20.953712 4945 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:20.915453 5075 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:20.931105 5218 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.014397 1968 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:21.017956 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 60306.9us
I20250626 08:24:21.009553 5094 raft_consensus.cc:695] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.014390 4945 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.026160 5113 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:21.021371 5036 consensus_queue.cc:1046] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.007s
I20250626 08:24:21.023417 5164 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.007s
I20250626 08:24:21.032532 5113 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:21.036976 5095 raft_consensus.cc:3053] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 LEADER]: Stepping down as leader of term 9
I20250626 08:24:21.037386 5095 raft_consensus.cc:738] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.037987 5075 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:21.039233 5193 log.cc:927] Time spent T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 1.549s user 0.003s sys 0.000s
I20250626 08:24:21.038327 5113 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:21.043215 5217 log.cc:927] Time spent T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.784s user 0.003s sys 0.001s
I20250626 08:24:21.047035 5135 raft_consensus.cc:3053] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Stepping down as leader of term 6
W20250626 08:24:21.047580 5216 log.cc:927] Time spent T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.786s user 0.003s sys 0.000s
I20250626 08:24:21.047668 5135 raft_consensus.cc:738] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.048687 5135 consensus_queue.cc:260] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.049695 5165 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.049862 5135 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.051280 5113 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.052649 5113 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.053001 5119 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.088238 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:21.090317 1841 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:21.096045 1843 raft_consensus.cc:2391] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:21.104599 1843 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1923) took 2162 ms (client timeout 2755 ms). Trace:
W20250626 08:24:21.108013 1847 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1821) took 1681 ms (client timeout 1924 ms). Trace:
W20250626 08:24:21.108127 1848 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1823) took 1646 ms (client timeout 2130 ms). Trace:
W20250626 08:24:21.109079 1847 rpcz_store.cc:269] 0626 08:24:19.426784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.039297 (+612513us) service_pool.cc:225] Handling call
0626 08:24:21.107541 (+1068244us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002bbbd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.107826 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":380377856}
W20250626 08:24:21.109174 1848 rpcz_store.cc:269] 0626 08:24:19.462007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.285000 (+822993us) service_pool.cc:225] Handling call
0626 08:24:21.107687 (+822687us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b54000348d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.107933 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43170176}
I20250626 08:24:21.112013 1848 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:21.112915 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 567) took 1253 ms. Trace:
I20250626 08:24:21.113322 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 607) took 1152 ms. Trace:
I20250626 08:24:21.113361 1848 rpcz_store.cc:276] 0626 08:24:19.859034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.110738 (+1251704us) service_pool.cc:225] Handling call
0626 08:24:21.111556 (+ 818us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:21.112542 (+ 986us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.112813 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17408}
I20250626 08:24:21.114074 1921 rpcz_store.cc:276] 0626 08:24:19.960392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.960580 (+ 188us) service_pool.cc:225] Handling call
0626 08:24:19.960993 (+ 413us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:19.988483 (+ 27490us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:19.988532 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:19.989205 (+ 673us) log.cc:844] Serialized 52 byte log entry
0626 08:24:20.120673 (+131468us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:20.120712 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.120772 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:21.041818 (+921046us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:21.112640 (+ 70822us) raft_consensus.cc:1683] finished
0626 08:24:21.112686 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.113086 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9878,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"spinlock_wait_cycles":31881472,"thread_start_us":131153,"threads_started":1,"wal-append.queue_time_us":269848}
I20250626 08:24:21.115989 1848 raft_consensus.cc:1238] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:21.116822 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 568) took 1243 ms. Trace:
I20250626 08:24:21.117252 1848 rpcz_store.cc:276] 0626 08:24:19.873297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.114996 (+1241699us) service_pool.cc:225] Handling call
0626 08:24:21.115627 (+ 631us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:21.116492 (+ 865us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.116738 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:21.118713 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:21.117259 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 108924us
W20250626 08:24:21.122438 1893 consensus_peers.cc:487] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 -> Peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Couldn't send request to peer 548099d81e7f46ac9b2f03d9e5e678a6. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60120 dropped due to backpressure: service queue is full with 50 items. This is attempt 1: this message will repeat every 5th retry.
W20250626 08:24:21.123006 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.29273e+06us
I20250626 08:24:21.124238 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:21.135499 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34646 (request call id 2254) took 2236 ms. Trace:
I20250626 08:24:21.136026 1998 rpcz_store.cc:276] 0626 08:24:18.898941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.086093 (+187152us) service_pool.cc:225] Handling call
0626 08:24:19.086348 (+ 255us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:19.440856 (+354508us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.134865 (+1694009us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54001671d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:21.135230 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":814067968}
I20250626 08:24:21.139415 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.142387 1921 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.143847 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.141466 1923 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20250626 08:24:21.145303 1923 pending_rounds.cc:85] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 5
I20250626 08:24:21.145591 1923 pending_rounds.cc:107] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250626 08:24:21.145882 1923 raft_consensus.cc:2887] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:20.991242 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1902) took 2725 ms. Trace:
I20250626 08:24:21.149021 1842 rpcz_store.cc:276] 0626 08:24:18.265399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.981677 (+2716278us) service_pool.cc:225] Handling call
0626 08:24:20.982084 (+ 407us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.983020 (+ 936us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:20.991189 (+ 8169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:21.144132 1928 raft_consensus.cc:1238] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:21.150978 1842 raft_consensus.cc:1273] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250626 08:24:21.139613 2551 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.160259 1925 raft_consensus.cc:1238] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 3. Current term is 4. Ops: []
I20250626 08:24:21.165118 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.166249 1925 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:21.168529 1928 raft_consensus.cc:1273] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250626 08:24:21.169588 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 712) took 1085 ms. Trace:
I20250626 08:24:21.170120 2581 rpcz_store.cc:276] 0626 08:24:20.084207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.084411 (+ 204us) service_pool.cc:225] Handling call
0626 08:24:20.084815 (+ 404us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:20.951870 (+867055us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.169156 (+217286us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b540050f354. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:21.169415 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1979231104}
I20250626 08:24:21.175323 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 691) took 2457 ms. Trace:
I20250626 08:24:21.176234 5007 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:21.177892 4911 raft_consensus.cc:3053] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:21.178351 4916 raft_consensus.cc:2747] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250626 08:24:21.182538 4959 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.183634 4959 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.186353 4960 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:21.187460 4960 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.178529 4911 raft_consensus.cc:738] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.042722 5094 consensus_queue.cc:237] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.204707 5189 raft_consensus.cc:2747] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250626 08:24:21.205411 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.206279 4979 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.175897 2583 rpcz_store.cc:276] 0626 08:24:18.717808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:18.717946 (+ 138us) service_pool.cc:225] Handling call
0626 08:24:18.718224 (+ 278us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:19.322048 (+603824us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.174654 (+1852606us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54003c0fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:21.175054 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1387237248}
I20250626 08:24:21.157496 1967 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:21.207897 1843 rpcz_store.cc:269] 0626 08:24:18.942030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.410851 (+1468821us) service_pool.cc:225] Handling call
0626 08:24:21.104215 (+693364us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b540032ebd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.104495 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1571976576}
W20250626 08:24:21.222311 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:21.222443 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.223440 1844 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:21.223747 5027 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250626 08:24:21.226469 5200 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Advancing to term 10
W20250626 08:24:21.227051 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.226704 5113 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250626 08:24:21.228295 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 710) took 1533 ms. Trace:
I20250626 08:24:21.228714 2585 rpcz_store.cc:276] 0626 08:24:19.695112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:19.695734 (+ 622us) service_pool.cc:225] Handling call
0626 08:24:21.227824 (+1532090us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54005bd954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.228075 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2723669760}
I20250626 08:24:21.230352 5114 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.231324 5114 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.232498 5063 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:21.234314 5021 consensus_queue.cc:1035] T 171f89014c824d17a510a3cc17b3af0d P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:21.038196 5095 consensus_queue.cc:260] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.207051 4979 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:21.251143 5230 log.cc:927] Time spent T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.425s user 0.004s sys 0.000s
I20250626 08:24:21.270020 5177 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:21.271338 5177 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.242393 5219 consensus_queue.cc:1046] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20250626 08:24:21.276333 4940 raft_consensus.cc:3053] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:21.276683 5177 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:21.276779 4940 raft_consensus.cc:738] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.278484 5114 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.229770 5113 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250626 08:24:21.253605 4719 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:21.258514 4911 consensus_queue.cc:260] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:21.260475 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 2.90265e+06us
W20250626 08:24:21.282191 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.255633 4959 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:21.237190 4918 consensus_queue.cc:1046] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20250626 08:24:21.283218 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 143972us
W20250626 08:24:21.298568 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 58677.3us
W20250626 08:24:21.300477 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 143402us
I20250626 08:24:21.302021 4973 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:21.302827 4973 raft_consensus.cc:695] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.304469 4973 consensus_queue.cc:237] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:21.306354 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 291893us
I20250626 08:24:21.307929 1968 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:21.310588 2549 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.313712 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:21.314100 5219 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:21.314836 5219 raft_consensus.cc:695] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:21.287134 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 163168us
I20250626 08:24:21.319541 1841 raft_consensus.cc:2466] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
W20250626 08:24:21.317143 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.338899 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1835) took 1510 ms. Trace:
I20250626 08:24:21.347724 1842 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250626 08:24:21.345134 1844 rpcz_store.cc:276] 0626 08:24:19.817389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.003473 (+1186084us) service_pool.cc:225] Handling call
0626 08:24:21.222246 (+218773us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:21.224042 (+ 1796us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.297113 (+ 73071us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540032ebd0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.327516 (+ 30403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477186304}
I20250626 08:24:21.358671 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 570) took 1079 ms. Trace:
I20250626 08:24:21.359241 1842 rpcz_store.cc:276] 0626 08:24:20.274914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.152054 (+877140us) service_pool.cc:225] Handling call
0626 08:24:21.284730 (+132676us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:21.348299 (+ 63569us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.354041 (+ 5742us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:21.359385 2581 raft_consensus.cc:1238] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:21.316262 5219 consensus_queue.cc:237] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.356436 4960 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.377763 1844 raft_consensus.cc:1238] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250626 08:24:21.385676 5113 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.386767 5095 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:21.386680 5113 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.358260 5177 consensus_queue.cc:1046] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20250626 08:24:21.389859 5177 raft_consensus.cc:3053] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:21.368197 4940 consensus_queue.cc:260] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.390612 5177 raft_consensus.cc:738] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.391665 4940 raft_consensus.cc:3058] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.392490 5177 consensus_queue.cc:260] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.390015 5051 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:21.400353 4945 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:21.394016 5119 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 11 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.399135 2578 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:21.399287 1850 raft_consensus.cc:1238] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 6. Ops: []
I20250626 08:24:21.400035 5051 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.393735 1842 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:60106 (request call id 995) took 1079 ms. Trace:
I20250626 08:24:21.403669 1842 rpcz_store.cc:276] 0626 08:24:20.314469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.362804 (+1048335us) service_pool.cc:225] Handling call
0626 08:24:21.393625 (+ 30821us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250626 08:24:21.402570 1742 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46332 (request call id 190) took 16807 ms (client timeout 15000 ms). Trace:
W20250626 08:24:21.407862 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 67799.8us
I20250626 08:24:21.413394 1849 raft_consensus.cc:2466] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:21.414423 5177 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.381680 5049 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.417558 5051 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.418494 4979 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:21.419421 1893 leader_election.cc:400] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:21.439927 1893 leader_election.cc:403] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250626 08:24:21.409642 5086 raft_consensus.cc:3053] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:21.423122 4959 consensus_queue.cc:1046] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20250626 08:24:21.424750 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 67741.2us
I20250626 08:24:21.372524 4911 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:21.421813 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6
I20250626 08:24:21.446221 1968 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.446683 1921 raft_consensus.cc:2466] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:21.442799 2551 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.452131 1845 raft_consensus.cc:1238] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20250626 08:24:21.454422 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:21.455349 5049 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:21.446187 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:21.458701 5248 log.cc:927] Time spent T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
W20250626 08:24:21.405404 1742 rpcz_store.cc:269] 0626 08:24:04.600114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:04.618116 (+ 18002us) service_pool.cc:225] Handling call
0626 08:24:21.402276 (+16784160us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:24:13.743798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:24:13.744490 (+ 692us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:24:13.744545 (+ 55us) write_op.cc:435] Acquired schema lock
0626 08:24:13.744581 (+ 36us) tablet.cc:660] Decoding operations
0626 08:24:13.923947 (+179366us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:24:13.939998 (+ 16051us) write_op.cc:641] Partition lock acquired for write op
0626 08:24:13.943744 (+ 3746us) tablet.cc:683] Acquiring locks for 17 operations
0626 08:24:14.028092 (+ 84348us) tablet.cc:699] Row locks acquired
0626 08:24:14.028129 (+ 37us) write_op.cc:260] PREPARE: finished
0626 08:24:14.028297 (+ 168us) write_op.cc:270] Start()
0626 08:24:14.028447 (+ 150us) write_op.cc:276] Timestamp: P: 1750926254028229 usec, L: 0
0626 08:24:14.028466 (+ 19us) op_driver.cc:348] REPLICATION: starting
0626 08:24:14.029247 (+ 781us) log.cc:844] Serialized 6728 byte log entry
0626 08:24:14.845260 (+816013us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb6102e6e 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611fd02 00007f4eb8cbd04b
0626 08:24:16.538030 (+1692770us) op_driver.cc:464] REPLICATION: finished
0626 08:24:16.777339 (+239309us) write_op.cc:301] APPLY: starting
0626 08:24:16.811400 (+ 34061us) tablet.cc:1365] starting BulkCheckPresence
0626 08:24:16.909434 (+ 98034us) tablet.cc:1368] finished BulkCheckPresence
0626 08:24:16.930457 (+ 21023us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:24:17.642815 (+712358us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:24:17.721988 (+ 79173us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0626 08:24:17.722081 (+ 93us) write_op.cc:312] APPLY: finished
0626 08:24:17.722694 (+ 613us) log.cc:844] Serialized 119 byte log entry
0626 08:24:18.036136 (+313442us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:18.052122 (+ 15986us) write_op.cc:454] Released schema lock
0626 08:24:18.071931 (+ 19809us) write_op.cc:341] FINISH: Updating metrics
0626 08:24:18.126061 (+ 54130us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:18.130221 (+ 4160us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2282735,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":121975,"apply.run_cpu_time_us":23347,"apply.run_wall_time_us":1373800,"num_ops":17,"prepare.queue_time_us":223177,"prepare.run_cpu_time_us":90902,"prepare.run_wall_time_us":2479689,"raft.queue_time_us":373200,"raft.run_cpu_time_us":15037,"raft.run_wall_time_us":127337,"replication_time_us":2496211,"spinlock_wait_cycles":102819200,"thread_start_us":2888097,"threads_started":5,"wal-append.queue_time_us":1372000,"wal-append.run_cpu_time_us":97919,"wal-append.run_wall_time_us":3389399}]]}
I20250626 08:24:21.441778 5086 raft_consensus.cc:738] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.467036 5086 consensus_queue.cc:260] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.468827 5086 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.471132 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1946) took 1155 ms. Trace:
I20250626 08:24:21.472082 2582 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 4. Current term is 5. Ops: []
I20250626 08:24:21.394362 4997 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:21.474138 4997 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.476668 4997 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.476959 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 714) took 1383 ms. Trace:
I20250626 08:24:21.477268 5051 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:21.478366 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.479118 1925 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:21.479053 1741 catalog_manager.cc:5582] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:24:21.484227 1921 pending_rounds.cc:85] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 3
I20250626 08:24:21.484622 1921 pending_rounds.cc:107] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250626 08:24:21.484886 1921 raft_consensus.cc:2887] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250626 08:24:21.486615 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 118075us
I20250626 08:24:21.486608 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.487900 5188 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:21.489239 5113 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.490301 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:21.492187 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:21.492802 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.477550 2578 rpcz_store.cc:276] 0626 08:24:20.093824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.097780 (+ 3956us) service_pool.cc:225] Handling call
0626 08:24:21.476744 (+1378964us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540044f054. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.476902 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2907954944}
I20250626 08:24:21.487563 1893 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:21.504523 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 566) took 1654 ms. Trace:
I20250626 08:24:21.506788 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1900) took 3073 ms. Trace:
I20250626 08:24:21.471665 1843 rpcz_store.cc:276] 0626 08:24:20.315245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.257959 (+942714us) service_pool.cc:225] Handling call
0626 08:24:21.258563 (+ 604us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:21.258972 (+ 409us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:21.258998 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:21.259531 (+ 533us) log.cc:844] Serialized 26 byte log entry
0626 08:24:21.340563 (+ 81032us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:21.363176 (+ 22613us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.374638 (+ 11462us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.374739 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:21.470825 (+ 96086us) raft_consensus.cc:1683] finished
0626 08:24:21.470863 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.471058 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30461,"prepare.queue_time_us":23009,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":28356,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":159,"spinlock_wait_cycles":185728,"thread_start_us":79917,"threads_started":1,"wal-append.queue_time_us":78537}
I20250626 08:24:21.525180 1926 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20250626 08:24:21.513096 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:21.538203 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 116516us
I20250626 08:24:21.506462 1847 rpcz_store.cc:276] 0626 08:24:19.850286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.110760 (+1260474us) service_pool.cc:225] Handling call
0626 08:24:21.111498 (+ 738us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:21.112209 (+ 711us) raft_consensus.cc:1532] Early marking committed up to index 10
0626 08:24:21.318585 (+206376us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.504001 (+185416us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.504184 (+ 183us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:21.504232 (+ 48us) raft_consensus.cc:1643] Marking committed up to 10
0626 08:24:21.504276 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.504348 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.504468 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":105763,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":64,"raft.queue_time_us":790,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":181,"spinlock_wait_cycles":20736,"thread_start_us":352349,"threads_started":2}
I20250626 08:24:21.540854 2001 raft_consensus.cc:1273] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250626 08:24:21.541281 5113 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:21.541937 5113 raft_consensus.cc:695] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
W20250626 08:24:21.542335 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 172685us
I20250626 08:24:21.542937 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.507292 1846 rpcz_store.cc:276] 0626 08:24:18.264846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.975107 (+2710261us) service_pool.cc:225] Handling call
0626 08:24:20.975383 (+ 276us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:20.975586 (+ 203us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:20.975610 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:20.976115 (+ 505us) log.cc:844] Serialized 26 byte log entry
0626 08:24:21.098562 (+122447us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:21.217640 (+119078us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.284231 (+ 66591us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.284328 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:21.288612 (+ 4284us) raft_consensus.cc:1683] finished
0626 08:24:21.288630 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.338297 (+ 49667us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3616,"prepare.queue_time_us":118789,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":60,"raft.queue_time_us":80432,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":174,"thread_start_us":240134,"threads_started":2,"wal-append.queue_time_us":62269}
I20250626 08:24:21.499454 5051 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.550509 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60120 (request call id 1832) took 1756 ms. Trace:
I20250626 08:24:21.551002 2584 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:21.551064 1845 rpcz_store.cc:276] 0626 08:24:19.794000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.000145 (+1206145us) service_pool.cc:225] Handling call
0626 08:24:21.451673 (+451528us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:21.452659 (+ 986us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.543424 (+ 90765us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54002562d0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.550422 (+ 6998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":979638528}
I20250626 08:24:21.552577 4943 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.552937 5200 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.553189 4943 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.553550 5200 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.554613 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.555765 5200 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.543946 1850 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 565) took 1813 ms. Trace:
I20250626 08:24:21.556699 1850 rpcz_store.cc:276] 0626 08:24:19.728996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:20.997880 (+1268884us) service_pool.cc:225] Handling call
0626 08:24:20.998171 (+ 291us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:21.440291 (+442120us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.541617 (+101326us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400210a54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:21.541820 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":869035392}
I20250626 08:24:21.558159 5075 raft_consensus.cc:491] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.558215 5227 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:21.558903 5075 raft_consensus.cc:513] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.559158 5214 consensus_queue.cc:1046] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20250626 08:24:21.561259 4959 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:21.561406 5075 leader_election.cc:290] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.522526 4979 raft_consensus.cc:3053] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Stepping down as leader of term 3
W20250626 08:24:21.552615 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36144 (request call id 680) took 3965 ms (client timeout 2104 ms). Trace:
W20250626 08:24:21.572714 2584 rpcz_store.cc:269] 0626 08:24:17.587532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:17.587710 (+ 178us) service_pool.cc:225] Handling call
0626 08:24:21.552371 (+3964661us) spinlock_profiling.cc:243] Waited 3.95 s on lock 0x7b54005429d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:21.552516 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9095180032}
I20250626 08:24:21.557654 5002 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:21.563961 5200 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.586740 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:21.591413 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.532199 4916 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.585886 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:21.592681 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:21.572245 4997 raft_consensus.cc:2747] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250626 08:24:21.647881 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:21.648411 1814 connection.cc:232] RPC call timeout handler was delayed by -1.910023689999889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20250626 08:24:21.597494 1924 raft_consensus.cc:2408] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20250626 08:24:21.595029 1846 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:21.592244 5188 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.600123 1919 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:21.601486 1927 raft_consensus.cc:1273] T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250626 08:24:21.602428 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 115021us
I20250626 08:24:21.606000 2587 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:21.561789 4959 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:21.606326 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.654150 4959 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.655093 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.655939 1923 raft_consensus.cc:2391] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:21.656821 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.543071 5113 consensus_queue.cc:237] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.607753 5051 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.608708 4943 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:21.609736 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 82461us
W20250626 08:24:21.610357 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 71296.6us
I20250626 08:24:21.610599 4973 consensus_queue.cc:1035] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250626 08:24:21.611274 5214 raft_consensus.cc:3053] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:21.606205 1850 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:21.617096 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.663061 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.667826 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.628561 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.629737 5095 consensus_queue.cc:1035] T 75222ddacb444f7ba27de92b1571ada8 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250626 08:24:21.634493 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 261241us
I20250626 08:24:21.635412 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.636039 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:21.636583 2551 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.639894 1741 catalog_manager.cc:5582] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 3, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 3 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
I20250626 08:24:21.643904 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:21.644567 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.646214 5189 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.572516 4979 raft_consensus.cc:738] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.670516 5214 raft_consensus.cc:738] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.671094 2001 raft_consensus.cc:2391] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 8: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:21.670920 2580 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:21.680303 1994 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:21.675246 1967 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:21.685307 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.687192 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:21.688220 2584 raft_consensus.cc:1273] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:21.690301 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.690780 1928 raft_consensus.cc:2391] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:21.691181 1919 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Advancing to term 11
I20250626 08:24:21.681648 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.693168 1923 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:21.694283 1741 catalog_manager.cc:5582] T 2e6a8a739b954cc082a290e6c529a271 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 4 to 5, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 5 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:24:21.700038 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.697978 4938 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.702055 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.702894 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.706249 1968 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.707398 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.707762 1968 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.708318 1924 raft_consensus.cc:2408] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250626 08:24:21.708587 4989 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.709236 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.710312 4938 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.710824 4979 consensus_queue.cc:260] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.711938 5109 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.712375 4979 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:21.712592 4938 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.712697 5109 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.710325 5189 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.713943 1814 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:21.715487 5189 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.677683 4916 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.718192 5119 raft_consensus.cc:2802] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:21.718570 1812 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:21.649205 5002 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.720072 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1944) took 1585 ms. Trace:
I20250626 08:24:21.721464 5002 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.720525 1848 rpcz_store.cc:276] 0626 08:24:20.134367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.118423 (+984056us) service_pool.cc:225] Handling call
0626 08:24:21.119045 (+ 622us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:21.120013 (+ 968us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:21.120057 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:21.221946 (+101889us) log.cc:844] Serialized 52 byte log entry
0626 08:24:21.443370 (+221424us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:21.502535 (+ 59165us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.517764 (+ 15229us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.523273 (+ 5509us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.523355 (+ 82us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:21.719850 (+196495us) raft_consensus.cc:1683] finished
0626 08:24:21.719872 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.720011 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33857,"prepare.queue_time_us":73556,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":3048,"raft.queue_time_us":95172,"raft.run_cpu_time_us":664,"raft.run_wall_time_us":83095,"thread_start_us":108457,"threads_started":1,"wal-append.queue_time_us":298251}
I20250626 08:24:21.671649 5200 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.722807 5030 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:21.722956 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 60457.2us
I20250626 08:24:21.723594 2550 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:21.713490 4662 consensus_meta.cc:287] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.861s user 0.016s sys 0.003s
I20250626 08:24:21.702003 1918 raft_consensus.cc:2466] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:21.727218 5229 raft_consensus.cc:3058] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.727313 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:21.728138 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 98287.8us
I20250626 08:24:21.704661 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.733057 1928 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.695668 5188 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.729969 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.709404 4989 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.754603 1925 raft_consensus.cc:1273] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:21.766551 2587 raft_consensus.cc:2391] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:21.767722 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.768441 2584 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:21.744243 1926 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:21.770220 5109 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.771042 1926 raft_consensus.cc:3058] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.771135 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.772233 5200 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.772318 5144 raft_consensus.cc:2747] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:21.772953 5162 consensus_queue.cc:1046] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20250626 08:24:21.774482 4956 raft_consensus.cc:3053] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Stepping down as leader of term 4
W20250626 08:24:21.774786 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 84728.2us
I20250626 08:24:21.774958 4956 raft_consensus.cc:738] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:21.775653 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 44611.4us
W20250626 08:24:21.777802 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.728662 4943 raft_consensus.cc:2802] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:24:21.787309 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 118543us
I20250626 08:24:21.784979 1893 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:21.774875 2551 leader_election.cc:304] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.704670 5219 raft_consensus.cc:2802] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:21.744375 1741 catalog_manager.cc:5582] T 4f29b6256843489397f6e58f34751764 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 3 to 6, leader changed from be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 6 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:24:21.721551 5055 consensus_queue.cc:1046] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20250626 08:24:21.793824 1924 raft_consensus.cc:2408] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250626 08:24:21.733355 5030 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:21.797816 5109 consensus_queue.cc:1046] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
W20250626 08:24:21.798374 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.798418 1927 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250626 08:24:21.799242 1927 pending_rounds.cc:85] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 2
I20250626 08:24:21.798738 5214 consensus_queue.cc:260] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.799480 1927 pending_rounds.cc:107] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250626 08:24:21.799747 1927 raft_consensus.cc:2887] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:21.800069 5214 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:21.801066 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.801795 1927 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:21.802748 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.803562 1927 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:21.803508 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.804435 1921 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:21.755244 5002 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.806525 5002 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.807548 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:21.785262 4956 consensus_queue.cc:260] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.808485 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1947) took 1439 ms. Trace:
I20250626 08:24:21.809082 4956 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:21.808873 1841 rpcz_store.cc:276] 0626 08:24:20.369108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:21.348390 (+979282us) service_pool.cc:225] Handling call
0626 08:24:21.386111 (+ 37721us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:21.429401 (+ 43290us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:21.429436 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:21.449350 (+ 19914us) log.cc:844] Serialized 26 byte log entry
0626 08:24:21.686473 (+237123us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:21.730414 (+ 43941us) log.cc:844] Serialized 14 byte log entry
0626 08:24:21.750381 (+ 19967us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:21.750476 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:21.808294 (+ 57818us) raft_consensus.cc:1683] finished
0626 08:24:21.808312 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:21.808425 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62551,"prepare.queue_time_us":41963,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":69923,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":17962,"thread_start_us":136748,"threads_started":1,"wal-append.queue_time_us":262327}
I20250626 08:24:21.811553 4989 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.814450 5002 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.816565 5109 raft_consensus.cc:2802] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:21.817150 5109 raft_consensus.cc:695] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.818415 5114 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:21.818645 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:21.819075 5007 raft_consensus.cc:3053] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:21.819509 5007 raft_consensus.cc:738] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.819681 4916 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:21.820398 1893 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:21.820510 5007 consensus_queue.cc:260] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.821317 5189 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:21.822187 5051 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:21.822798 5051 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
W20250626 08:24:21.823609 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 115980us
I20250626 08:24:21.823789 5051 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.824735 1968 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:21.821867 5189 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.825624 4495 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:21.829098 4495 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:21.829445 4495 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.818184 5109 consensus_queue.cc:237] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.841028 2550 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:21.843451 5021 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:21.843392 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:21.844651 1993 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.819026 5114 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:21.846616 5114 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.849459 1967 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:21.849747 4719 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:21.850724 5075 raft_consensus.cc:3053] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:21.851287 5075 raft_consensus.cc:738] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:21.839402 5007 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.852363 5075 consensus_queue.cc:260] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.843989 5021 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:21.854314 5021 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.858389 1893 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:21.859498 5201 raft_consensus.cc:2747] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250626 08:24:21.861091 1893 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:21.851023 4719 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.862025 5201 raft_consensus.cc:2747] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:21.862466 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:21.863521 4719 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.863930 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:21.864809 1998 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Advancing to term 11
I20250626 08:24:21.864763 1741 catalog_manager.cc:5582] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 2, leader changed from 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 2 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 08:24:21.832556 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.866514 1999 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 4.
I20250626 08:24:21.829703 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.868674 1993 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250626 08:24:21.868960 4495 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:21.869194 1741 catalog_manager.cc:5582] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 4, leader changed from 755d83ca0913406b816e95415d502055 (127.0.171.195) to 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193). New cstate: current_term: 4 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
I20250626 08:24:21.870445 1923 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:21.871383 4495 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:21.871899 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 47259.1us
I20250626 08:24:21.874701 4916 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:21.833307 4997 raft_consensus.cc:2762] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20250626 08:24:21.880002 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.887964 1925 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:21.880715 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:21.881419 4911 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:21.863400 1893 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:21.834877 5188 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.887336 4916 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:21.890731 1924 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:21.881322 5189 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:21.733551 4662 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:21.895041 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 54133.3us
I20250626 08:24:21.898371 1919 raft_consensus.cc:2408] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20250626 08:24:21.899681 4719 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:21.900243 4719 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:21.900696 4719 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.840615 5030 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.892431 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:21.903075 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:21.884400 4943 raft_consensus.cc:695] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:21.885500 5219 raft_consensus.cc:695] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.912176 4911 raft_consensus.cc:695] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [term 7 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:21.912237 4943 consensus_queue.cc:237] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.913568 4911 consensus_queue.cc:237] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.914932 5219 consensus_queue.cc:237] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.902030 4916 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.922070 5189 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:21.920923 1741 catalog_manager.cc:5582] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 4 to 5, leader changed from be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 5 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 08:24:21.929327 4719 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:21.932776 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:21.933444 1924 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:21.936501 1968 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.936833 4719 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:21.938457 1741 catalog_manager.cc:5582] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } }
W20250626 08:24:21.939548 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:21.940009 5162 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:21.941365 1968 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:21.885207 5119 raft_consensus.cc:695] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:21.951772 4495 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:21.959954 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:21.958022 1741 catalog_manager.cc:5582] T 331d5bdb3fda41489445725e41b94bd1 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 4, leader changed from be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 4 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250626 08:24:21.963861 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 100575us
I20250626 08:24:21.965029 2000 raft_consensus.cc:1238] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 6. Current term is 8. Ops: []
I20250626 08:24:21.955185 5189 raft_consensus.cc:695] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:21.977154 1993 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:21.980538 1741 catalog_manager.cc:5582] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 reported cstate change: term changed from 2 to 4, leader changed from 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194) to 755d83ca0913406b816e95415d502055 (127.0.171.195). New cstate: current_term: 4 leader_uuid: "755d83ca0913406b816e95415d502055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } health_report { overall_health: HEALTHY } } }
W20250626 08:24:21.989078 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 272128us
W20250626 08:24:22.033813 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:22.035880 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:22.038122 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 341353us
W20250626 08:24:22.039953 1816 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.050509 1893 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:24:22.002056 5086 consensus_meta.cc:287] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.533s user 0.016s sys 0.000s
I20250626 08:24:22.059959 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:22.062288 1812 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:21.979308 4959 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:22.079991 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:22.088409 1968 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:22.093251 5162 raft_consensus.cc:2747] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:22.094764 5229 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:22.070322 5119 consensus_queue.cc:237] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.138252 4940 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:22.097440 5114 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.139370 5177 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:22.139905 5177 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.141377 4662 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.142374 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:22.143110 5177 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:22.144280 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:22.143129 1921 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.143766 5114 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.100688 2000 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:22.101773 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:22.106107 2551 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:22.106860 5086 consensus_queue.cc:1035] T 0899df8ec53c45498dd1f3da415d5155 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250626 08:24:22.149080 2551 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:22.149873 4719 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:22.150187 5270 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:22.150731 5270 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.150099 5113 raft_consensus.cc:3058] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.151158 5270 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:22.100037 4959 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.114068 5269 consensus_queue.cc:1046] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.040s
W20250626 08:24:22.115041 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 129151us
I20250626 08:24:22.115939 1814 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
W20250626 08:24:22.137109 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 48577.5us
I20250626 08:24:22.122099 4938 consensus_queue.cc:1046] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:22.124572 5189 consensus_queue.cc:237] T f2bf9541d0634f9fab914afd74bb9505 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.128293 1926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:22.159559 1926 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.128698 1919 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:22.133980 4495 consensus_queue.cc:1035] T 5f31173b902a46949ba221ed157ca482 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250626 08:24:22.134965 4979 consensus_queue.cc:1035] T 94ba4a836fec402199d1aecf04c04b62 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:22.145846 1994 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:22.076678 4997 raft_consensus.cc:2747] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:22.111806 5075 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:22.167037 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:22.157197 1924 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:22.163425 1998 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 11.
I20250626 08:24:22.165484 1925 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:22.168201 1928 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:22.169835 2550 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:22.172012 1967 leader_election.cc:304] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.176321 1993 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:22.178316 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:22.182408 1923 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250626 08:24:22.176832 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:22.172092 1893 leader_election.cc:336] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Remote error: Timed out: Call waited in the queue past client deadline
I20250626 08:24:22.184090 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:22.195142 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 46139.5us
I20250626 08:24:22.195961 2551 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:22.191515 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:22.206503 1997 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
I20250626 08:24:22.207278 1967 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:22.184046 5109 raft_consensus.cc:2802] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:22.215950 2584 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:22.194763 2000 raft_consensus.cc:2466] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:22.219769 1996 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250626 08:24:22.225080 4719 raft_consensus.cc:2747] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250626 08:24:22.225983 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.230015 5220 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:24:22.234802 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 71492.8us
W20250626 08:24:22.235862 1893 leader_election.cc:336] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Remote error: Timed out: Call waited in the queue past client deadline
W20250626 08:24:22.237057 2548 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.238159 2584 raft_consensus.cc:1273] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:22.241940 5271 consensus_queue.cc:1035] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20250626 08:24:22.243852 5271 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:22.245204 5269 raft_consensus.cc:3053] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:22.194080 5162 raft_consensus.cc:2802] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:22.179910 5021 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:22.247974 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 51550.6us
I20250626 08:24:22.179210 5177 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
W20250626 08:24:22.255940 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 60090.8us
I20250626 08:24:22.257125 5214 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:22.261711 1923 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:22.262101 1921 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:22.263697 2550 leader_election.cc:304] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:22.264357 4956 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:22.264977 4956 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.265754 4943 raft_consensus.cc:2802] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:22.266348 4943 raft_consensus.cc:695] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:22.267197 4956 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.267313 4943 consensus_queue.cc:237] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.268126 4916 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:22.269362 5279 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250626 08:24:22.269762 4719 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250626 08:24:22.271054 1999 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:22.262038 4911 consensus_queue.cc:1035] T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250626 08:24:22.276856 5220 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:22.277321 5220 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.279616 4495 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:22.280181 4495 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.282510 5270 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.285144 5270 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:22.257575 5162 raft_consensus.cc:695] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:22.287484 5162 consensus_queue.cc:237] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.288851 687 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 08:24:22.249490 5109 raft_consensus.cc:695] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:22.241518 5030 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.290948 5109 consensus_queue.cc:237] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.292317 5030 leader_election.cc:290] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:22.293167 5030 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:22.293619 5030 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.294070 5030 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.299291 5021 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:22.245630 5269 raft_consensus.cc:738] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:22.301438 5269 consensus_queue.cc:260] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.302287 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:22.302824 4495 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.302963 1921 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:22.303135 5269 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:22.301551 2551 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:22.305583 2551 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:22.308384 1926 raft_consensus.cc:2466] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:22.313611 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:22.317375 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.317276 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:22.319058 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 82036.8us
I20250626 08:24:22.321841 1893 leader_election.cc:400] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:22.322785 1893 leader_election.cc:403] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250626 08:24:22.323436 4997 raft_consensus.cc:2747] T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7.
I20250626 08:24:22.323966 5220 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:22.321617 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 191318us
I20250626 08:24:22.325613 1893 leader_election.cc:400] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 8. Current term is 9.
I20250626 08:24:22.326113 1893 leader_election.cc:403] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250626 08:24:22.326431 5220 leader_election.cc:290] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.330718 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:22.331372 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 210964us
I20250626 08:24:22.350953 1967 leader_election.cc:304] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.326840 4997 raft_consensus.cc:2747] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 8. Current term is 9.
I20250626 08:24:22.355978 5271 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:22.364060 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:22.331050 5021 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:22.373464 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:22.368003 2587 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:22.380089 1927 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.374176 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:22.374434 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:22.375044 5271 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:22.375998 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.381819 1923 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.384408 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:22.381103 2582 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 10.
I20250626 08:24:22.350952 4719 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:22.384120 1816 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:22.395278 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 69863.2us
I20250626 08:24:22.396409 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:22.370723 4911 raft_consensus.cc:2802] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
W20250626 08:24:22.403468 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 287657us
I20250626 08:24:22.405040 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.353153 5086 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20250626 08:24:22.370533 4916 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:22.412916 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.413151 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.411554 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:22.415872 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.416230 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.416800 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.392343 4719 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.411182 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.417380 1846 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:22.402040 5113 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:22.423668 1816 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.418782 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.432904 1848 raft_consensus.cc:1273] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250626 08:24:22.417969 5075 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:22.445564 1841 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:22.424427 4719 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:22.448515 1843 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.449208 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.421819 4911 raft_consensus.cc:695] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:22.456915 1844 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:22.459084 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.439174 5002 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:22.458032 1816 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.466655 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.470583 1841 raft_consensus.cc:3058] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.465171 1847 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:22.475099 4496 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:22.446195 5114 raft_consensus.cc:2747] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250626 08:24:22.476420 1926 raft_consensus.cc:3053] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:22.476817 1926 raft_consensus.cc:738] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:22.477662 1926 consensus_queue.cc:260] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.479027 1926 raft_consensus.cc:3058] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.491160 5030 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.491613 5271 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:22.492116 2581 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:22.492393 5002 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.494861 5002 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:22.493052 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 78449.2us
I20250626 08:24:22.475593 4911 consensus_queue.cc:237] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:22.501875 1923 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:22.504120 1927 raft_consensus.cc:2466] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:22.512568 1843 raft_consensus.cc:2466] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:22.529934 5286 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 11 FOLLOWER]: Leader election won for term 11
I20250626 08:24:22.530565 5286 raft_consensus.cc:695] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [term 11 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:22.530926 1842 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:22.531533 5286 consensus_queue.cc:237] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.532451 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.533450 1842 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:22.534598 1844 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:22.475605 4496 raft_consensus.cc:491] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:22.529700 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 71762.5us
I20250626 08:24:22.534879 1846 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250626 08:24:22.544049 1841 raft_consensus.cc:2466] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:22.551187 1926 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:22.555294 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.587323 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.588301 1847 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
W20250626 08:24:22.592877 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 94347.2us
I20250626 08:24:22.553714 4496 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:22.552153 5021 raft_consensus.cc:695] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:22.505621 5119 consensus_queue.cc:1046] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20250626 08:24:22.637456 1814 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:22.611932 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:22.634509 1816 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.635844 1846 raft_consensus.cc:1273] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250626 08:24:22.647387 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.648281 1843 raft_consensus.cc:3058] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:22.653049 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 242494us
I20250626 08:24:22.657837 1967 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.659060 1967 leader_election.cc:304] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.660378 1847 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:22.665050 4662 raft_consensus.cc:2802] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:22.665457 5055 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:22.673159 5055 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.673579 5055 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.673163 1926 raft_consensus.cc:1273] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:22.665848 4662 raft_consensus.cc:695] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:22.680821 4662 consensus_queue.cc:237] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.682240 4959 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250626 08:24:22.669463 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:22.670116 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:22.684240 1999 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:22.684468 1996 raft_consensus.cc:2466] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:22.671636 5119 raft_consensus.cc:3053] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:22.685721 5119 raft_consensus.cc:738] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:22.686918 5119 consensus_queue.cc:260] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:22.688781 5119 raft_consensus.cc:3058] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:22.690747 1816 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:22.691824 5291 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:22.692294 5291 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:22.692669 5291 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:22.715885 5055 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.610041 5095 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
W20250626 08:24:22.720829 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 64649.7us
I20250626 08:24:22.735064 1843 raft_consensus.cc:2466] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:22.755071 1999 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:22.718307 4938 raft_consensus.cc:2747] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:22.719939 4956 raft_consensus.cc:2802] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:22.756572 1816 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:22.774372 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.723915 5109 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:22.783145 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:22.783855 1846 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:22.786548 2578 raft_consensus.cc:1273] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20250626 08:24:22.790822 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 175741us
I20250626 08:24:22.662442 4997 consensus_queue.cc:1046] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.105s
I20250626 08:24:22.796216 1814 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:22.807283 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:22.786130 5296 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20250626 08:24:22.820590 1843 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250626 08:24:22.824084 1847 raft_consensus.cc:1273] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:22.786648 4956 raft_consensus.cc:695] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:22.837113 4956 consensus_queue.cc:237] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.838155 5270 raft_consensus.cc:3053] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 08:24:22.838716 5270 raft_consensus.cc:738] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:22.839802 5270 consensus_queue.cc:260] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.796195 5113 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:22.833220 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:22.825196 5021 consensus_queue.cc:237] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.754344 5291 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:22.912142 5270 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:22.912760 1999 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:22.825824 5119 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 08:24:22.821427 4938 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:22.926407 1847 raft_consensus.cc:1273] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:22.927522 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:22.832023 5030 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:22.928306 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:22.909019 5272 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250626 08:24:22.792610 5109 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:22.957355 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:22.959843 1921 raft_consensus.cc:3053] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 08:24:22.960317 1921 raft_consensus.cc:738] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:22.961344 1921 consensus_queue.cc:260] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.963328 1921 raft_consensus.cc:3058] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:22.975606 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:22.953081 5269 consensus_meta.cc:287] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.649s user 0.012s sys 0.001s
I20250626 08:24:22.941133 5086 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:22.954098 5293 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250626 08:24:22.907681 4911 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250626 08:24:22.917212 5113 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.997148 1850 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:22.941396 5007 consensus_queue.cc:1035] T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250626 08:24:22.935519 5002 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:22.989301 5086 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:23.058097 1927 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:23.061863 1999 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:23.003687 4938 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250626 08:24:23.071929 1921 raft_consensus.cc:1273] T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250626 08:24:23.003057 5269 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.084465 1999 raft_consensus.cc:1238] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: []
W20250626 08:24:23.066448 5307 log.cc:927] Time spent T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
W20250626 08:24:23.087266 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 159347us
W20250626 08:24:23.090482 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 298359us
I20250626 08:24:23.067991 5086 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:23.091646 1814 leader_election.cc:304] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
W20250626 08:24:23.094534 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 111441us
I20250626 08:24:23.073282 4938 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:23.106797 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.074045 5055 leader_election.cc:290] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.090384 5119 raft_consensus.cc:695] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:23.081446 5269 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.112557 4938 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:23.128189 1999 raft_consensus.cc:1215] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 3.6->[5.7-7.8] Dedup: 7.8->[]
I20250626 08:24:23.135246 1999 raft_consensus.cc:1238] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20250626 08:24:23.131784 1968 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:23.133251 1844 raft_consensus.cc:1238] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250626 08:24:23.137476 1841 raft_consensus.cc:1238] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 6. Current term is 7. Ops: [3.5-6.6]
W20250626 08:24:23.128894 4719 consensus_meta.cc:287] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.666s user 0.013s sys 0.000s
I20250626 08:24:23.140322 4719 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.142462 4719 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:23.145618 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 54061.3us
I20250626 08:24:23.117434 5007 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:23.154098 2580 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:23.139307 1968 leader_election.cc:304] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:23.167517 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 222962us
I20250626 08:24:23.176163 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.170050 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:23.183854 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:23.151125 5291 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.150697 5055 raft_consensus.cc:2747] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:23.288403 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 121378us
W20250626 08:24:23.289300 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 142119us
I20250626 08:24:23.301451 1814 leader_election.cc:304] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:23.130662 5299 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.195s
I20250626 08:24:23.154356 5329 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:23.306522 5331 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:23.315371 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 213794us
I20250626 08:24:23.176012 5314 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:23.316960 5302 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
I20250626 08:24:23.167809 5007 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:23.347453 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:23.351269 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.371309 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:23.375972 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.377449 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.352655 5113 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.319049 5331 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:23.285508 5317 consensus_queue.cc:1035] T 5e2fc654780843abbda82ba8f5df515e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.339s
I20250626 08:24:23.393746 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:23.407675 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 106373us
I20250626 08:24:23.428112 5331 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:23.428582 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:23.372684 5055 raft_consensus.cc:2747] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250626 08:24:23.409569 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 92870.2us
I20250626 08:24:23.440567 1841 raft_consensus.cc:2466] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:23.330428 5095 consensus_queue.cc:1035] T d606b2ba0bd74eea8cd6e0c6e109c5ac P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.062s
W20250626 08:24:23.396384 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 96182us
I20250626 08:24:23.335888 4997 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.425040 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:23.336328 5109 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.353189 5002 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:23.495756 5002 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:24:23.442613 1968 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:23.434517 5305 log.cc:927] Time spent T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.400s user 0.001s sys 0.001s
I20250626 08:24:23.492578 1844 raft_consensus.cc:2366] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:24:23.493335 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.392441 4911 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20250626 08:24:23.499106 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.380029 5314 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:23.528398 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45835.9us
I20250626 08:24:23.383760 5326 consensus_queue.cc:1035] T 7a16389239e54c52bb619a62b1b0b91a P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.192s
I20250626 08:24:23.555622 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:23.556571 1846 consensus_meta.cc:287] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.765s user 0.010s sys 0.001s
I20250626 08:24:23.552160 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36142 (request call id 772) took 1185 ms. Trace:
I20250626 08:24:23.561992 1846 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:23.563827 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.565874 1846 raft_consensus.cc:2366] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
W20250626 08:24:23.498704 5303 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
W20250626 08:24:23.498466 5318 log.cc:927] Time spent T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.348s user 0.001s sys 0.000s
I20250626 08:24:23.561637 2583 rpcz_store.cc:276] 0626 08:24:22.367290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.367454 (+ 164us) service_pool.cc:225] Handling call
0626 08:24:22.389519 (+ 22065us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.396674 (+ 7155us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.396722 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.400659 (+ 3937us) log.cc:844] Serialized 26 byte log entry
0626 08:24:22.503846 (+103187us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:22.503886 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:22.503960 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:23.551938 (+1047978us) raft_consensus.cc:1683] finished
0626 08:24:23.551963 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:23.552099 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11130,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8480,"thread_start_us":63057,"threads_started":1,"wal-append.queue_time_us":185086}
I20250626 08:24:23.345692 5119 consensus_queue.cc:237] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:23.549235 1813 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.035s sys 0.051s
W20250626 08:24:23.613010 1813 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.035s sys 0.051s
I20250626 08:24:23.616974 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.480556 5299 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:23.480269 5075 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
I20250626 08:24:23.336840 5329 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.477486 5328 consensus_queue.cc:1035] T 6f8cb31b10924fbcb39dfbbcac77397e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.511s
W20250626 08:24:23.647085 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:23.705090 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.717427 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1961) took 1298 ms. Trace:
I20250626 08:24:23.679636 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:23.701720 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:23.707116 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 264596us
W20250626 08:24:23.700632 1926 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.008s sys 0.043s
W20250626 08:24:23.719539 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:23.720057 1926 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s user 0.008s sys 0.043s
W20250626 08:24:23.604800 5307 log.cc:927] Time spent T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20250626 08:24:23.719590 1841 raft_consensus.cc:2391] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:23.724239 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.726152 1846 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 8.
I20250626 08:24:23.647547 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:23.605870 5269 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:23.717962 1845 rpcz_store.cc:276] 0626 08:24:22.418887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.429019 (+ 10132us) service_pool.cc:225] Handling call
0626 08:24:22.445580 (+ 16561us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.465469 (+ 19889us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.465509 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.474486 (+ 8977us) log.cc:844] Serialized 26 byte log entry
0626 08:24:22.724223 (+249737us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:23.032318 (+308095us) log.cc:844] Serialized 14 byte log entry
0626 08:24:23.032524 (+ 206us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:23.032619 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:23.717166 (+684547us) raft_consensus.cc:1683] finished
0626 08:24:23.717194 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:23.717353 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46503,"prepare.queue_time_us":227186,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":5714,"raft.queue_time_us":139094,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":13739,"thread_start_us":498519,"threads_started":2,"wal-append.queue_time_us":482939}
I20250626 08:24:23.741768 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:23.715417 5301 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.392s user 0.001s sys 0.001s
I20250626 08:24:23.559897 4997 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.750047 1846 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:23.716676 4911 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:23.821118 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 74715.6us
I20250626 08:24:23.712954 5341 raft_consensus.cc:2802] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election won for term 8
W20250626 08:24:23.825703 5086 consensus_meta.cc:287] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.708s user 0.009s sys 0.000s
W20250626 08:24:23.746778 5315 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.370s user 0.000s sys 0.001s
W20250626 08:24:23.826148 5296 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250626 08:24:23.829339 1921 raft_consensus.cc:2391] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:23.746728 5336 raft_consensus.cc:2747] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:23.749342 5326 raft_consensus.cc:2802] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250626 08:24:23.816208 5303 log.cc:927] Time spent T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20250626 08:24:23.743279 5007 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:23.829843 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 78580us
I20250626 08:24:23.828482 5338 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.374s
W20250626 08:24:23.863601 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 428606us
I20250626 08:24:23.859906 5329 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:23.876641 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1959) took 1482 ms. Trace:
W20250626 08:24:23.876699 5309 log.cc:927] Time spent T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.512s user 0.001s sys 0.000s
I20250626 08:24:23.876814 5361 consensus_queue.cc:1035] T 9996474aab3b4a5ca0e10b9f200b2abc P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.128s
I20250626 08:24:23.877089 4997 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.860101 5326 raft_consensus.cc:491] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:23.880579 5189 raft_consensus.cc:3053] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:23.852913 5086 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.918318 1846 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250626 08:24:23.882303 5332 consensus_queue.cc:1035] T 148557a00636474cad8673c1f02aae5e P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.171s
I20250626 08:24:23.858089 5345 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:23.735812 5299 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:23.934363 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.015s sys 0.015s
I20250626 08:24:23.899646 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:23.934406 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:23.934880 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.015s sys 0.015s
I20250626 08:24:23.905427 1814 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:23.910822 5342 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:23.911667 5219 raft_consensus.cc:491] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.847265 4911 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.941898 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:23.942622 2584 raft_consensus.cc:2466] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:24:23.943013 5335 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250626 08:24:23.900369 1849 rpcz_store.cc:276] 0626 08:24:22.394458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.398523 (+ 4065us) service_pool.cc:225] Handling call
0626 08:24:22.418704 (+ 20181us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.430742 (+ 12038us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.430765 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.436422 (+ 5657us) log.cc:844] Serialized 26 byte log entry
0626 08:24:22.548098 (+111676us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:22.753800 (+205702us) log.cc:844] Serialized 14 byte log entry
0626 08:24:22.770358 (+ 16558us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:22.770657 (+ 299us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:23.876430 (+1105773us) raft_consensus.cc:1683] finished
0626 08:24:23.876451 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:23.876581 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2369,"prepare.queue_time_us":276818,"prepare.run_cpu_time_us":6846,"prepare.run_wall_time_us":75392,"raft.queue_time_us":28332,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":10113,"spinlock_wait_cycles":53888,"thread_start_us":254662,"threads_started":2,"wal-append.queue_time_us":346221}
I20250626 08:24:23.943290 4911 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:23.929946 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 108028us
I20250626 08:24:23.944844 4911 consensus_queue.cc:1046] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250626 08:24:23.746793 5333 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.445s
I20250626 08:24:23.945775 5219 raft_consensus.cc:513] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.946381 5333 raft_consensus.cc:3053] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:23.946758 5333 raft_consensus.cc:738] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
I20250626 08:24:23.861373 5341 raft_consensus.cc:695] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [term 8 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:23.947922 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.007s sys 0.038s
I20250626 08:24:23.947651 5333 consensus_queue.cc:260] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:23.948319 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.007s sys 0.038s
I20250626 08:24:23.935096 2578 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:23.942028 5351 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.916960 5189 raft_consensus.cc:738] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:23.938675 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 2024) took 1632 ms. Trace:
I20250626 08:24:23.952277 1812 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:23.882875 5346 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:23.919982 5326 raft_consensus.cc:3058] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:23.948755 5341 consensus_queue.cc:237] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.960929 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.956198 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:23.955029 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.001s sys 0.011s
I20250626 08:24:23.961019 1999 raft_consensus.cc:3058] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:23.962782 2001 raft_consensus.cc:2391] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:23.955619 1919 rpcz_store.cc:276] 0626 08:24:22.305849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.306391 (+ 542us) service_pool.cc:225] Handling call
0626 08:24:22.327354 (+ 20963us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.358626 (+ 31272us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.358685 (+ 59us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.393649 (+ 34964us) log.cc:844] Serialized 26 byte log entry
0626 08:24:22.611288 (+217639us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:22.611334 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:22.611408 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:23.937671 (+1326263us) raft_consensus.cc:1683] finished
0626 08:24:23.938489 (+ 818us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:23.938621 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":264530,"raft.queue_time_us":20845,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":21200,"thread_start_us":183386,"threads_started":1,"wal-append.queue_time_us":520360}
I20250626 08:24:23.963975 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1964) took 1496 ms. Trace:
W20250626 08:24:23.964352 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.002s sys 0.012s
I20250626 08:24:23.921152 4997 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:23.964483 1848 rpcz_store.cc:276] 0626 08:24:22.467676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.467855 (+ 179us) service_pool.cc:225] Handling call
0626 08:24:22.495348 (+ 27493us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.538077 (+ 42729us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.538127 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.538663 (+ 536us) log.cc:844] Serialized 26 byte log entry
0626 08:24:22.718483 (+179820us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:22.818942 (+100459us) log.cc:844] Serialized 14 byte log entry
0626 08:24:22.820996 (+ 2054us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:22.821086 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:23.963714 (+1142628us) raft_consensus.cc:1683] finished
0626 08:24:23.963744 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:23.963905 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46411,"prepare.queue_time_us":216914,"prepare.run_wall_time_us":54,"raft.queue_time_us":212579,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":38648,"spinlock_wait_cycles":15094784,"thread_start_us":250741,"threads_started":2,"wal-append.queue_time_us":590842}
I20250626 08:24:23.967093 1849 raft_consensus.cc:1238] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250626 08:24:23.969931 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 73148us
I20250626 08:24:23.971303 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:23.972465 1846 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:23.975397 4495 raft_consensus.cc:491] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.975616 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:23.976034 4495 raft_consensus.cc:513] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.976509 2002 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 4.
W20250626 08:24:23.971001 5270 consensus_meta.cc:287] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.041s user 0.011s sys 0.002s
W20250626 08:24:23.969661 4496 consensus_meta.cc:287] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 1.318s user 0.009s sys 0.005s
I20250626 08:24:23.968914 5314 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:23.960618 5219 leader_election.cc:290] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:23.992904 2551 leader_election.cc:304] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:23.971280 5075 raft_consensus.cc:2747] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:23.996778 5330 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.996810 4496 raft_consensus.cc:513] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:23.996973 4495 leader_election.cc:290] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:23.982434 4997 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.953878 5342 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.023010 5364 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.023749 1816 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:24.024866 5356 log.cc:927] Time spent T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250626 08:24:24.026013 5329 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.026077 5363 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.985456 5361 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.026597 5362 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:23.966018 5346 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:23.951493 5345 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.027444 5299 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:24.028646 5331 consensus_meta.cc:287] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.600s user 0.011s sys 0.001s
W20250626 08:24:24.030758 5353 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250626 08:24:24.031872 5332 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 74a4b66e1f7b4aed8fae26d31f9cc4e7)
I20250626 08:24:24.032565 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75222ddacb444f7ba27de92b1571ada8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.033577 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.033531 1918 raft_consensus.cc:1238] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:24.033694 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:24.000104 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 75501.6us
I20250626 08:24:24.038908 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9122458a1eae4f07b1a7f4ff63d6be19" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:24.189742 1924 raft_consensus.cc:2366] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8.
I20250626 08:24:24.190577 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:24.191787 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.192262 5372 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.192674 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:24.193239 1924 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:24.193019 5372 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.004043 1842 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:23.957865 5351 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.195284 5372 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.196305 2583 raft_consensus.cc:2408] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250626 08:24:24.196516 5351 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:24.000250 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.197988 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.198206 5375 raft_consensus.cc:491] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:24.198873 5375 raft_consensus.cc:513] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.199309 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.200115 1848 raft_consensus.cc:2391] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 8: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:24.200910 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.029716 5086 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:24.050040 5330 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.046108 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:23.996966 5358 raft_consensus.cc:2802] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:24.206497 5358 raft_consensus.cc:491] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.206940 5358 raft_consensus.cc:3058] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:24.208077 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.208719 1918 raft_consensus.cc:2466] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:24.210490 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:24.210851 4496 leader_election.cc:290] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:24.212431 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 2031) took 1527 ms. Trace:
I20250626 08:24:24.163934 5370 raft_consensus.cc:491] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.026243 5339 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.024s
W20250626 08:24:24.052659 4938 consensus_meta.cc:287] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.879s user 0.008s sys 0.004s
I20250626 08:24:24.128769 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:24.173411 5338 raft_consensus.cc:491] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.153734 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:24.154681 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:24.046118 5189 consensus_queue.cc:260] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.242364 2578 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:24.242154 2550 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:24.156270 1847 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
I20250626 08:24:24.161839 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a65a55472e64be4b9ae56426278c3c1" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:24.248963 2001 raft_consensus.cc:3058] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
W20250626 08:24:24.163476 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 171268us
I20250626 08:24:24.250269 2551 leader_election.cc:304] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:24.210736 5368 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:24.162889 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 222036us
W20250626 08:24:24.166546 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 142806us
I20250626 08:24:24.166880 1997 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:24.169974 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.170143 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:24.255105 2000 raft_consensus.cc:3058] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:24.172727 1999 raft_consensus.cc:1273] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:24.177217 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.203868 1920 raft_consensus.cc:2391] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 6: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:24.203369 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.203361 5330 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.126928 1925 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250626 08:24:24.034003 5291 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.011s sys 0.009s
I20250626 08:24:24.231271 4496 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.225315 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.130360 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.273598 1924 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:24.261114 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.286916 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 743) took 1842 ms. Trace:
I20250626 08:24:24.176586 5361 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.212090 5349 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:24.213012 1926 rpcz_store.cc:276] 0626 08:24:22.685337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.685584 (+ 247us) service_pool.cc:225] Handling call
0626 08:24:22.685969 (+ 385us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:22.741858 (+ 55889us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:22.741901 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:22.787762 (+ 45861us) log.cc:844] Serialized 26 byte log entry
0626 08:24:23.720644 (+932882us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:23.720685 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:23.720747 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:24.212229 (+491482us) raft_consensus.cc:1683] finished
0626 08:24:24.212252 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:24.212381 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35789,"raft.queue_time_us":62787,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11955,"thread_start_us":817390,"threads_started":1,"wal-append.queue_time_us":1073991}
I20250626 08:24:24.223098 5030 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.287969 1924 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250626 08:24:24.265985 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.223510 5375 leader_election.cc:290] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:24.187669 5329 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.308164 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:24.316027 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 53139.5us
I20250626 08:24:24.316912 1893 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:24.317317 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.317994 1848 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:24.319952 1925 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20250626 08:24:24.320951 1925 pending_rounds.cc:85] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 5
I20250626 08:24:24.321339 1925 pending_rounds.cc:107] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20250626 08:24:24.321729 1925 raft_consensus.cc:2887] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:24.268580 2551 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:24.269588 1812 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:24.287328 2581 rpcz_store.cc:276] 0626 08:24:22.444907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.445069 (+ 162us) service_pool.cc:225] Handling call
0626 08:24:22.456639 (+ 11570us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:22.494417 (+ 37778us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:24.286655 (+1792238us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b54005bd954. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:24.286836 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3788672}
W20250626 08:24:24.303393 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 48006.5us
I20250626 08:24:24.299120 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:24.316332 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.238713 5342 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:24.176142 5364 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.181382 5363 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.267405 5299 raft_consensus.cc:2802] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:24.187397 5331 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.350026 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0914c29163894ada9540f1133a461648" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.333392 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:24.356366 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:24.291630 5291 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s user 0.016s sys 0.012s
I20250626 08:24:24.295563 1993 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
I20250626 08:24:24.259688 5327 raft_consensus.cc:2802] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:24.242094 4938 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.212294 5362 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.296591 5314 consensus_queue.cc:1046] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.164s
I20250626 08:24:24.223012 5332 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.245426 5338 raft_consensus.cc:513] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:24.503257 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 240096us
I20250626 08:24:24.381407 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.454450 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.536932 1848 raft_consensus.cc:2466] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:24.540180 2002 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 4: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:24.257151 5370 raft_consensus.cc:513] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.268090 5368 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.558563 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.561658 1995 raft_consensus.cc:2408] T 6f8cb31b10924fbcb39dfbbcac77397e P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250626 08:24:24.567499 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:24.568307 1921 raft_consensus.cc:2391] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 10: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:24.539482 5021 raft_consensus.cc:2747] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250626 08:24:24.302345 4496 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.317788 5030 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.383113 5341 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:24.534643 5327 raft_consensus.cc:491] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:24.605754 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.606304 1847 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:24.298012 5270 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.270s
I20250626 08:24:24.592038 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.545374 4997 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.613070 1925 raft_consensus.cc:2408] T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:24.613481 4997 raft_consensus.cc:2747] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:24.590507 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.615478 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:24.616511 1848 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
W20250626 08:24:24.594429 1814 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:24.566690 4989 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:24.455049 5372 consensus_queue.cc:1035] T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.125s
I20250626 08:24:24.575677 5219 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:24.575870 5326 consensus_meta.cc:287] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.606s user 0.010s sys 0.001s
I20250626 08:24:24.572985 5375 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:24.611812 1994 raft_consensus.cc:1273] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:24.605902 5362 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:24.624532 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 307684us
W20250626 08:24:24.607693 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:24.630970 1998 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250626 08:24:24.605659 4938 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.665261 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.665978 1995 raft_consensus.cc:2466] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 10.
I20250626 08:24:24.608451 5327 raft_consensus.cc:3058] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Advancing to term 9
W20250626 08:24:24.621587 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 78588.8us
W20250626 08:24:24.677606 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 218987us
I20250626 08:24:24.666766 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:24.629998 5359 log.cc:927] Time spent T 4be9d96212a6463eb1aa71d0b3d300a6 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.404s user 0.000s sys 0.001s
I20250626 08:24:24.677630 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.678467 1994 raft_consensus.cc:2466] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:24.679307 1920 raft_consensus.cc:2408] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250626 08:24:24.685240 1925 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250626 08:24:24.685485 1893 leader_election.cc:304] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:24.636631 5379 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:24.691633 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 422125us
I20250626 08:24:24.608691 5341 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.706413 5381 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:24.688534 5335 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250626 08:24:24.660573 5346 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.745604 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.734234 4989 raft_consensus.cc:695] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:24.754452 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:24.754627 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:24.821280 2578 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 10.
I20250626 08:24:24.687549 5270 consensus_queue.cc:1230] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer 755d83ca0913406b816e95415d502055 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250626 08:24:24.755653 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:24.756126 5349 raft_consensus.cc:695] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:24.757115 2579 raft_consensus.cc:1238] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:24.758244 2582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:24.758705 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:24.759357 2585 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
W20250626 08:24:24.686654 5289 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.734s user 0.031s sys 0.044s
I20250626 08:24:24.737490 2000 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:24.791484 1924 raft_consensus.cc:1273] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250626 08:24:24.792658 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:24.658893 5219 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.674073 5345 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:24.806593 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "187f08b11130495ba8e2a482370447f4" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.659258 5326 raft_consensus.cc:513] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.666078 5375 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.826545 5346 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:24.748893 5379 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.713138 5299 raft_consensus.cc:695] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Becoming Leader. State: Replica: be56082cacb54e9aa8f1a9ee91e70d76, State: Running, Role: LEADER
I20250626 08:24:24.820353 5381 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
I20250626 08:24:24.806074 5351 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.839699 5361 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:24.826156 5329 leader_election.cc:290] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:24.845769 2584 raft_consensus.cc:2466] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:24.850422 1994 raft_consensus.cc:2466] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:24.854647 2582 raft_consensus.cc:2391] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 8: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:24.857396 1924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.846551 2586 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:24.886485 1848 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9.
W20250626 08:24:24.847033 5289 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.906s user 0.033s sys 0.046s
I20250626 08:24:24.892058 1968 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:24.905452 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.906515 1995 raft_consensus.cc:2391] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 9: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:24.913228 1996 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 9: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:24.919055 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:24.934365 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f29b6256843489397f6e58f34751764" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.949355 5414 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250626 08:24:24.949995 5378 log.cc:927] Time spent T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.280s user 0.000s sys 0.001s
I20250626 08:24:24.942858 4989 consensus_queue.cc:237] T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:24.951656 5369 log.cc:927] Time spent T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.362s user 0.001s sys 0.000s
I20250626 08:24:24.952113 5392 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.952332 5409 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:24.952677 5392 raft_consensus.cc:513] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.952912 5409 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.949940 5414 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:24.911499 5341 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:24.961634 2585 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 10.
I20250626 08:24:24.904892 5351 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:24.873324 5346 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:24.845203 5349 consensus_queue.cc:237] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:24.957093 5360 log.cc:927] Time spent T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.486s user 0.000s sys 0.001s
I20250626 08:24:24.955797 5400 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:24.985741 5279 log.cc:927] Time spent T 4f29b6256843489397f6e58f34751764 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 2.599s user 0.011s sys 0.000s
I20250626 08:24:24.991799 5351 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:24.992182 5374 log.cc:927] Time spent T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.514s user 0.001s sys 0.001s
I20250626 08:24:24.992537 4496 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.992089 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:24.994195 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1976) took 1094 ms. Trace:
I20250626 08:24:24.959228 5030 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:24.994617 1841 rpcz_store.cc:276] 0626 08:24:23.900054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.900183 (+ 129us) service_pool.cc:225] Handling call
0626 08:24:23.917299 (+ 17116us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:23.937396 (+ 20097us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:23.939069 (+ 1673us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:23.940898 (+ 1829us) log.cc:844] Serialized 26 byte log entry
0626 08:24:24.094930 (+154032us) raft_consensus.cc:1643] Marking committed up to 3
0626 08:24:24.170661 (+ 75731us) log.cc:844] Serialized 14 byte log entry
0626 08:24:24.175380 (+ 4719us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:24.175481 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:24.993971 (+818490us) raft_consensus.cc:1683] finished
0626 08:24:24.993992 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:24.994134 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29759,"prepare.queue_time_us":407772,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":15167,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":16414,"thread_start_us":134408,"threads_started":1,"wal-append.queue_time_us":283854}
I20250626 08:24:24.966949 5414 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:25.014366 1927 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250626 08:24:25.015703 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:25.010278 2001 consensus_meta.cc:287] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.759s user 0.011s sys 0.000s
I20250626 08:24:24.983253 5331 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:24.994650 5376 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.438s user 0.001s sys 0.001s
I20250626 08:24:24.994658 5417 raft_consensus.cc:2747] T cd7130b92c4840deaca578e523eb4a09 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:25.016533 1927 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:25.027782 5219 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.028673 5346 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.028971 5381 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:24.996344 5364 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.043969 1846 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 591) took 1064 ms. Trace:
W20250626 08:24:25.046079 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.033658 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:24.983073 5368 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:25.038705 5387 log.cc:927] Time spent T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.393s user 0.001s sys 0.000s
W20250626 08:24:25.042155 5385 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
I20250626 08:24:25.034664 5370 leader_election.cc:290] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.034798 5299 consensus_queue.cc:237] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.032379 5394 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:25.053817 5399 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.054072 5394 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.054451 5399 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.003968 5338 leader_election.cc:290] T 94ba4a836fec402199d1aecf04c04b62 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.000873 5363 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.047232 1927 raft_consensus.cc:2466] T 3946d60648894026984085c3abee51ea P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:25.061203 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.044493 1846 rpcz_store.cc:276] 0626 08:24:23.979565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.979727 (+ 162us) service_pool.cc:225] Handling call
0626 08:24:23.993839 (+ 14112us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:24.028349 (+ 34510us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.028391 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:24.029041 (+ 650us) log.cc:844] Serialized 52 byte log entry
0626 08:24:24.187389 (+158348us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:24.316394 (+129005us) log.cc:844] Serialized 14 byte log entry
0626 08:24:24.317072 (+ 678us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:24.317207 (+ 135us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:25.043709 (+726502us) raft_consensus.cc:1683] finished
0626 08:24:25.043730 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:25.043898 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38614,"prepare.queue_time_us":331657,"raft.queue_time_us":222935,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":13050,"spinlock_wait_cycles":19943808,"thread_start_us":228178,"threads_started":2,"wal-append.queue_time_us":269746}
W20250626 08:24:25.034637 5377 log.cc:927] Time spent T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.494s user 0.002s sys 0.000s
I20250626 08:24:25.022578 2001 raft_consensus.cc:2466] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 8.
I20250626 08:24:25.073369 1927 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:25.081660 1846 raft_consensus.cc:1273] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20250626 08:24:25.089643 1994 raft_consensus.cc:1238] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:25.071559 5363 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.009093 5400 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.098766 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:25.100250 1967 leader_election.cc:304] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:25.100718 1927 raft_consensus.cc:3058] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:25.074857 5392 leader_election.cc:290] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.096108 4496 raft_consensus.cc:491] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.104518 5372 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 08:24:25.104868 4496 raft_consensus.cc:513] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.124989 1967 leader_election.cc:304] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:25.134064 1919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 648) took 1142 ms. Trace:
I20250626 08:24:25.134665 1919 rpcz_store.cc:276] 0626 08:24:23.991330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.991638 (+ 308us) service_pool.cc:225] Handling call
0626 08:24:24.021650 (+ 30012us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:24.047231 (+ 25581us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.154752 (+107521us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:24.165301 (+ 10549us) log.cc:844] Serialized 26 byte log entry
0626 08:24:24.378271 (+212970us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:24.378335 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:24.378436 (+ 101us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:25.131716 (+753280us) raft_consensus.cc:1683] finished
0626 08:24:25.131747 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:25.133974 (+ 2227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18022,"raft.queue_time_us":159266,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":43256,"thread_start_us":163390,"threads_started":1,"wal-append.queue_time_us":444217}
I20250626 08:24:25.103317 5332 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.123312 5341 raft_consensus.cc:2747] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:25.139119 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.139622 4496 leader_election.cc:290] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.140841 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:25.141295 5189 raft_consensus.cc:3058] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:25.141554 1919 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:25.143070 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.144412 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.145025 1919 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 3.
I20250626 08:24:25.146358 1843 raft_consensus.cc:3058] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:25.093858 5363 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.100577 5326 leader_election.cc:290] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:25.164134 1850 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1952) took 2765 ms (client timeout 2007 ms). Trace:
W20250626 08:24:25.165731 1850 rpcz_store.cc:269] 0626 08:24:22.380462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.389109 (+ 8647us) service_pool.cc:225] Handling call
0626 08:24:25.146020 (+2756911us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b5400180454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.146123 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1359738112}
I20250626 08:24:25.164194 5214 raft_consensus.cc:2747] T 187f08b11130495ba8e2a482370447f4 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:25.201277 1843 raft_consensus.cc:1273] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250626 08:24:25.205943 1927 raft_consensus.cc:2466] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9.
I20250626 08:24:25.212432 1919 raft_consensus.cc:1273] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:25.179370 5379 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:25.228224 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.223661 5200 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
W20250626 08:24:25.229569 2580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 773) took 2847 ms (client timeout 2738 ms). Trace:
I20250626 08:24:25.230068 5399 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:25.232317 5415 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250626 08:24:25.246207 5200 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.248526 5200 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.229614 5400 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:25.243831 2580 rpcz_store.cc:269] 0626 08:24:22.382717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:22.382886 (+ 169us) service_pool.cc:225] Handling call
0626 08:24:25.229281 (+2846395us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b5400605ed4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.229478 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1724930432}
I20250626 08:24:25.314312 5409 leader_election.cc:290] T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.316256 5393 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.316968 5393 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.331864 1845 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 594) took 1351 ms. Trace:
W20250626 08:24:25.336589 5367 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 1.102s user 0.003s sys 0.000s
I20250626 08:24:25.226099 5394 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:25.351781 1921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39254 (request call id 2150) took 1320 ms (client timeout 1575 ms). Trace:
W20250626 08:24:25.354382 1921 rpcz_store.cc:269] 0626 08:24:24.031625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.033402 (+ 1777us) service_pool.cc:225] Handling call
0626 08:24:25.351508 (+1318106us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54002dfad4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.351675 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":871974144}
I20250626 08:24:25.361783 4997 raft_consensus.cc:3053] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 08:24:25.362326 4997 raft_consensus.cc:738] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:25.363240 4997 consensus_queue.cc:260] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:25.364323 4997 raft_consensus.cc:3058] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:25.368184 5363 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.406443 5375 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:25.311200 5358 consensus_meta.cc:287] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 1.104s user 0.008s sys 0.005s
I20250626 08:24:25.407575 5358 raft_consensus.cc:513] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.411346 5358 leader_election.cc:290] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.419536 1844 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 588) took 2250 ms. Trace:
I20250626 08:24:25.430040 1844 rpcz_store.cc:276] 0626 08:24:23.169066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.731696 (+562630us) service_pool.cc:225] Handling call
0626 08:24:23.748198 (+ 16502us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:23.817944 (+ 69746us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:23.817990 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:23.850680 (+ 32690us) log.cc:844] Serialized 26 byte log entry
0626 08:24:23.968629 (+117949us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:23.968666 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:23.968727 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:25.335662 (+1366935us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:25.419155 (+ 83493us) raft_consensus.cc:1683] finished
0626 08:24:25.419186 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:25.419458 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8847,"raft.queue_time_us":70731,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":21613,"spinlock_wait_cycles":10861312,"thread_start_us":117470,"threads_started":1,"wal-append.queue_time_us":316255}
I20250626 08:24:25.341692 1845 rpcz_store.cc:276] 0626 08:24:23.980487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.980602 (+ 115us) service_pool.cc:225] Handling call
0626 08:24:23.994716 (+ 14114us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:24.029710 (+ 34994us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.029740 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:24.030466 (+ 726us) log.cc:844] Serialized 78 byte log entry
0626 08:24:24.180409 (+149943us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:24.269926 (+ 89517us) log.cc:844] Serialized 14 byte log entry
0626 08:24:24.294801 (+ 24875us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.030530 (+735729us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:25.331209 (+300679us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b18003d9300. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611d555 00007f4eb9076c50 00007f4eb90892f6 00007f4eb9099e67
0626 08:24:25.331396 (+ 187us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.331501 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:25.331562 (+ 61us) raft_consensus.cc:1683] finished
0626 08:24:25.331583 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:25.331780 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47285,"prepare.queue_time_us":851000,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":84,"raft.queue_time_us":228377,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":23794,"spinlock_wait_cycles":138729856,"thread_start_us":268835,"threads_started":3,"wal-append.queue_time_us":275731}
I20250626 08:24:25.435163 1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60136 (request call id 1986) took 1231 ms. Trace:
I20250626 08:24:25.435706 1847 rpcz_store.cc:276] 0626 08:24:24.203884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.265607 (+ 61723us) service_pool.cc:225] Handling call
0626 08:24:25.434927 (+1169320us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540032f0d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.435086 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707521920}
I20250626 08:24:25.443187 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36142 (request call id 778) took 1529 ms. Trace:
I20250626 08:24:25.443737 2583 rpcz_store.cc:276] 0626 08:24:23.913577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.914536 (+ 959us) service_pool.cc:225] Handling call
0626 08:24:25.442908 (+1528372us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54004362d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.443102 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":599660800}
I20250626 08:24:25.445693 5333 raft_consensus.cc:3058] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:25.488346 1926 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.047s sys 0.071s
W20250626 08:24:25.490280 1926 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s user 0.047s sys 0.071s
W20250626 08:24:25.491859 1966 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.033s sys 0.068s
W20250626 08:24:25.492178 1966 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.781s user 0.033s sys 0.068s
W20250626 08:24:25.508180 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.024s sys 0.056s
W20250626 08:24:25.508549 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.024s sys 0.056s
W20250626 08:24:25.503669 5327 consensus_meta.cc:287] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.816s user 0.012s sys 0.001s
W20250626 08:24:25.518684 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.520072 1814 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:25.524253 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:25.525298 1997 raft_consensus.cc:2391] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:25.525753 1997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1841) took 1105 ms. Trace:
W20250626 08:24:25.533586 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.528153 1997 rpcz_store.cc:276] 0626 08:24:24.420895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.521331 (+1100436us) service_pool.cc:225] Handling call
0626 08:24:25.525686 (+ 4355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250626 08:24:25.521482 5271 raft_consensus.cc:491] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.576712 5271 raft_consensus.cc:513] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.578934 5271 leader_election.cc:290] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.574609 5328 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.580336 5328 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.558939 5177 raft_consensus.cc:491] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:25.583200 1928 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2154) took 1550 ms. Trace:
I20250626 08:24:25.583979 1814 leader_election.cc:400] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8.
I20250626 08:24:25.584616 1814 leader_election.cc:403] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250626 08:24:25.587694 1814 leader_election.cc:304] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:25.583611 1928 rpcz_store.cc:276] 0626 08:24:24.032835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.034056 (+ 1221us) service_pool.cc:225] Handling call
0626 08:24:24.034331 (+ 275us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:24.235324 (+200993us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.235372 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:24.245141 (+ 9769us) raft_consensus.cc:1643] Marking committed up to 4
0626 08:24:24.245204 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.582966 (+1337762us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400260554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:25.583012 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:25.583141 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":442406272}
I20250626 08:24:25.588961 5271 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813) with higher term. Message: Invalid argument: T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8.
I20250626 08:24:25.614058 1843 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1973) took 2523 ms. Trace:
I20250626 08:24:25.614635 1843 rpcz_store.cc:276] 0626 08:24:23.090402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.097511 (+ 7109us) service_pool.cc:225] Handling call
0626 08:24:25.146219 (+2048708us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:25.202005 (+ 55786us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.613788 (+411783us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400180450. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.613981 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4710656256}
I20250626 08:24:25.581565 5177 raft_consensus.cc:513] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:25.618492 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.589284 5284 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:25.520308 5327 raft_consensus.cc:513] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.522607 5414 raft_consensus.cc:513] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:25.636309 1924 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.009s sys 0.040s
W20250626 08:24:25.640511 1924 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s user 0.010s sys 0.040s
I20250626 08:24:25.640604 1993 raft_consensus.cc:1238] T 2717a332547d4ee8912bdda5311cffd8 P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Rejecting Update request from peer 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 8. Current term is 9. Ops: []
I20250626 08:24:25.640776 1995 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 4.
W20250626 08:24:25.637533 2549 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.032s sys 0.079s
W20250626 08:24:25.641710 2549 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.752s user 0.032s sys 0.080s
W20250626 08:24:25.643812 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2303) took 1512 ms (client timeout 1657 ms). Trace:
W20250626 08:24:25.644336 1996 rpcz_store.cc:269] 0626 08:24:24.128365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.128509 (+ 144us) service_pool.cc:225] Handling call
0626 08:24:25.640149 (+1511640us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b540039c454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.640301 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1557326208}
W20250626 08:24:25.672215 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 84643.5us
W20250626 08:24:25.673542 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:25.671663 1918 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.135s user 0.018s sys 0.051s
W20250626 08:24:25.682734 1918 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.147s user 0.019s sys 0.051s
W20250626 08:24:25.678104 1849 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.411s user 0.057s sys 0.054s
W20250626 08:24:25.684296 1849 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.417s user 0.057s sys 0.055s
I20250626 08:24:25.670135 5358 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:25.671965 5177 leader_election.cc:290] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:25.679706 2548 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.056s sys 0.109s
I20250626 08:24:25.682479 5324 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:25.687422 5375 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:25.729782 2002 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1834) took 1635 ms (client timeout 1618 ms). Trace:
W20250626 08:24:25.730307 2002 rpcz_store.cc:269] 0626 08:24:24.094435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.128637 (+ 34202us) service_pool.cc:225] Handling call
0626 08:24:25.729534 (+1600897us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54001671d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.729688 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510785280}
W20250626 08:24:25.718333 2548 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.801s user 0.057s sys 0.109s
I20250626 08:24:25.741549 1998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 504) took 1571 ms. Trace:
I20250626 08:24:25.744120 1998 rpcz_store.cc:276] 0626 08:24:24.170244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.554443 (+384199us) service_pool.cc:225] Handling call
0626 08:24:24.554869 (+ 426us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:24.660947 (+106078us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.741292 (+1080345us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b5400203fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:25.741479 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174094208}
I20250626 08:24:25.719355 5328 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.712255 5291 consensus_queue.cc:1046] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20250626 08:24:25.755553 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:25.755656 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.020s sys 0.048s
I20250626 08:24:25.719036 5002 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
W20250626 08:24:25.758778 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.013s user 0.020s sys 0.048s
I20250626 08:24:25.726415 5358 consensus_queue.cc:1035] T b498813b859f40cd95ed6adf6a2d8918 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20250626 08:24:25.758800 1925 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250626 08:24:25.731048 5375 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.721459 5324 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:25.768785 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.098s user 0.014s sys 0.049s
I20250626 08:24:25.743775 5336 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20250626 08:24:25.771822 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.105s user 0.014s sys 0.049s
W20250626 08:24:25.773516 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.756420 5418 raft_consensus.cc:491] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.776664 5418 raft_consensus.cc:513] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:25.760497 5338 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:25.774513 5412 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:25.775043 1998 raft_consensus.cc:1273] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20250626 08:24:25.775416 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:25.775837 5327 leader_election.cc:290] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.775859 1996 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:25.769440 5414 leader_election.cc:290] T 766debd72f3f4feaa9e81cd3690afdc1 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 4 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.760839 5328 raft_consensus.cc:491] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.783797 5418 leader_election.cc:290] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:25.779572 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:25.878813 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:25.880084 2001 raft_consensus.cc:1273] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250626 08:24:25.793201 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.005s sys 0.019s
W20250626 08:24:25.880775 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.014s user 0.005s sys 0.019s
W20250626 08:24:25.793462 1890 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.009s sys 0.012s
W20250626 08:24:25.881600 1890 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.092s user 0.009s sys 0.013s
W20250626 08:24:25.792800 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.386s user 0.004s sys 0.020s
W20250626 08:24:25.898218 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.540s user 0.005s sys 0.020s
W20250626 08:24:25.898563 5299 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.032s sys 0.018s
W20250626 08:24:25.899046 5299 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s user 0.032s sys 0.018s
W20250626 08:24:25.898955 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.63049e+06us
I20250626 08:24:25.899914 2585 raft_consensus.cc:1238] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 LEADER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:25.900079 2551 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:25.793875 1841 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.012s sys 0.011s
W20250626 08:24:25.901286 1841 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.012s sys 0.012s
W20250626 08:24:25.792466 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.003s sys 0.020s
I20250626 08:24:25.839411 2002 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
W20250626 08:24:25.902211 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.192s user 0.004s sys 0.021s
W20250626 08:24:25.902598 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 1.1926e+06us
I20250626 08:24:25.878813 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.880019 1997 raft_consensus.cc:2466] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
W20250626 08:24:25.785370 1842 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60120 (request call id 1836) took 2693 ms (client timeout 1618 ms). Trace:
I20250626 08:24:25.905790 2002 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20250626 08:24:25.907262 1842 rpcz_store.cc:269] 0626 08:24:23.091697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.091831 (+ 134us) service_pool.cc:225] Handling call
0626 08:24:25.785208 (+2693377us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400211e54. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.785305 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2061361024}
I20250626 08:24:25.770395 5300 raft_consensus.cc:491] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:25.869017 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:25.890479 5423 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.911154 5423 raft_consensus.cc:513] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.894829 5418 consensus_queue.cc:1035] T 187f08b11130495ba8e2a482370447f4 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250626 08:24:25.914650 5300 raft_consensus.cc:513] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.917451 5300 leader_election.cc:290] T f9b805acc8ae4c3abfe6d6ee40fbf456 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:25.913829 5423 leader_election.cc:290] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.918802 5423 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.919540 5423 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:25.897806 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:25.923023 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.923295 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.923738 5352 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:25.924100 1928 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 3.
I20250626 08:24:25.924512 5352 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.926312 1814 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:25.929026 5352 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.932175 1890 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:25.934664 5352 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250626 08:24:25.935664 5352 raft_consensus.cc:491] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:25.936568 5352 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Advancing to term 11
W20250626 08:24:25.942761 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.945960 1890 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:25.879619 1920 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.006s sys 0.023s
W20250626 08:24:25.949105 1920 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.007s sys 0.023s
I20250626 08:24:25.951757 2582 raft_consensus.cc:1273] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250626 08:24:25.953938 5355 raft_consensus.cc:2802] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:25.954411 5355 raft_consensus.cc:491] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.954797 5355 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:25.955271 1999 raft_consensus.cc:1215] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20250626 08:24:25.955792 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:25.957583 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1849) took 1003 ms. Trace:
I20250626 08:24:25.955816 1999 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250626 08:24:25.958089 1994 rpcz_store.cc:276] 0626 08:24:24.954340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.954843 (+ 503us) service_pool.cc:225] Handling call
0626 08:24:25.089399 (+134556us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:25.095432 (+ 6033us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.957361 (+861929us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54003a6950. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:25.957513 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265546624}
I20250626 08:24:25.962281 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:25.870150 5227 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.870671 4938 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:25.966792 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.966684 5227 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.966887 4938 raft_consensus.cc:513] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:25.968032 5390 raft_consensus.cc:491] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.960868 1996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:25.885507 5270 raft_consensus.cc:491] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:25.972642 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.950855 5422 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:25.973425 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:25.968874 5227 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:25.972906 1965 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:25.992712 1965 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:25.983469 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 83508us
I20250626 08:24:25.859136 5338 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.968559 5390 raft_consensus.cc:513] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:25.998898 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.019511 1927 raft_consensus.cc:2466] T 331d5bdb3fda41489445725e41b94bd1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 7.
I20250626 08:24:26.014827 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.960983 5002 raft_consensus.cc:491] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:25.988528 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.275s user 0.045s sys 0.060s
I20250626 08:24:25.995158 2551 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
W20250626 08:24:26.064548 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.352s user 0.047s sys 0.066s
I20250626 08:24:26.021672 1919 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 6.
I20250626 08:24:25.888386 5328 raft_consensus.cc:513] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } }
W20250626 08:24:26.079212 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 84138.3us
W20250626 08:24:25.986193 5349 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.020s sys 0.007s
I20250626 08:24:25.973824 4938 leader_election.cc:290] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:26.086349 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:25.989192 5291 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:25.989028 5368 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250626 08:24:26.095980 1967 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.007s sys 0.014s
W20250626 08:24:26.084061 1968 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 1.879s (SENT) [suppressed 8 similar messages]
I20250626 08:24:25.980583 5414 consensus_queue.cc:1046] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20250626 08:24:26.112605 1919 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:26.059849 5423 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:26.102418 1967 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.967s user 0.008s sys 0.015s
I20250626 08:24:26.063251 4662 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:26.118055 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:26.122224 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.45278e+06us
I20250626 08:24:26.068871 5227 raft_consensus.cc:2747] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:25.989543 5270 raft_consensus.cc:513] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:26.093080 5286 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.014s sys 0.007s
I20250626 08:24:25.886214 5412 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.091279 5002 raft_consensus.cc:513] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:26.080359 5342 raft_consensus.cc:491] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:26.131493 1893 leader_election.cc:400] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:24:26.136019 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:26.090180 5300 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250626 08:24:26.016773 5299 consensus_queue.cc:1035] T 7e7d84adc07945fe8a3aa16f71b111e3 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
W20250626 08:24:26.095794 5349 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.063s user 0.022s sys 0.009s
I20250626 08:24:26.002935 5422 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:26.111016 5289 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.102s user 0.043s sys 0.098s
W20250626 08:24:26.433194 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 209114us
I20250626 08:24:26.439617 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.522097 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.446061 1893 leader_election.cc:403] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20250626 08:24:26.128222 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 993226us
W20250626 08:24:26.285773 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:26.356921 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.143138 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:26.523630 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.524073 1927 raft_consensus.cc:2466] T 4be9d96212a6463eb1aa71d0b3d300a6 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
W20250626 08:24:26.524216 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.458017 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.523439 1967 leader_election.cc:304] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:26.359547 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.390239 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:26.348062 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.168761 2551 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:26.103487 5368 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:26.099529 5291 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 08:24:26.538355 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 406953us
I20250626 08:24:26.390597 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:26.541218 2586 raft_consensus.cc:2466] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:26.540395 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d606b2ba0bd74eea8cd6e0c6e109c5ac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
W20250626 08:24:26.539788 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.523393 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.537827 2578 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.586681 1893 leader_election.cc:304] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:26.347195 5429 log.cc:927] Time spent T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
W20250626 08:24:26.348359 5424 log.cc:927] Time spent T 6f8cb31b10924fbcb39dfbbcac77397e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.391s user 0.001s sys 0.000s
I20250626 08:24:26.587615 2580 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 9.
W20250626 08:24:26.545267 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 376629us
I20250626 08:24:26.624689 2551 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:26.590539 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:26.626341 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.601945 1919 raft_consensus.cc:1273] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250626 08:24:26.170603 4989 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.214s user 0.029s sys 0.062s
W20250626 08:24:26.194885 5436 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20250626 08:24:26.194202 5425 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
W20250626 08:24:26.611091 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 87739.1us
W20250626 08:24:26.446879 5286 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.187s user 0.030s sys 0.020s
W20250626 08:24:26.587509 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.538105 5390 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:26.521962 5289 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.517s user 0.050s sys 0.127s
W20250626 08:24:26.634189 4989 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.681s user 0.037s sys 0.068s
I20250626 08:24:26.194021 5420 raft_consensus.cc:491] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:26.352789 5324 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:26.644801 1919 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250626 08:24:26.224550 5437 raft_consensus.cc:491] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
W20250626 08:24:26.632385 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 106395us
I20250626 08:24:26.682969 1967 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:26.345851 5352 raft_consensus.cc:513] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.634694 5393 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:26.130437 4662 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:26.696643 1890 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:26.542157 5433 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250626 08:24:26.708626 2587 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s user 0.025s sys 0.085s
W20250626 08:24:26.747254 2587 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.137s user 0.025s sys 0.085s
I20250626 08:24:26.622346 5331 raft_consensus.cc:2747] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:26.343708 5355 raft_consensus.cc:513] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.524092 5414 consensus_queue.cc:1035] T 70ce63b4769e4ca8980ddb20229389ef P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.229s
I20250626 08:24:26.644698 5300 raft_consensus.cc:2802] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250626 08:24:26.682999 1848 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.688s user 0.056s sys 0.043s
I20250626 08:24:26.538719 5342 raft_consensus.cc:513] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.726320 1967 leader_election.cc:304] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:26.699425 5214 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.495s user 0.001s sys 0.034s
I20250626 08:24:26.734699 5333 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250626 08:24:26.743356 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 46795.5us
I20250626 08:24:26.708141 5426 raft_consensus.cc:2747] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250626 08:24:26.146996 5327 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.585s
W20250626 08:24:26.814374 1848 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.820s user 0.059s sys 0.061s
I20250626 08:24:26.768585 1849 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41946 (request call id 592) took 2788 ms. Trace:
I20250626 08:24:26.776115 1921 raft_consensus.cc:2391] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 11: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:26.708375 5323 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20250626 08:24:26.834434 1928 raft_consensus.cc:2391] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:26.838245 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34656 (request call id 1837) took 2676 ms. Trace:
I20250626 08:24:26.749593 4662 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:26.621148 5291 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
W20250626 08:24:26.816637 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 90398.6us
I20250626 08:24:26.848989 1967 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:26.814512 5300 raft_consensus.cc:491] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:26.840564 2585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 758) took 1053 ms. Trace:
I20250626 08:24:26.855269 2585 rpcz_store.cc:276] 0626 08:24:25.787020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.787204 (+ 184us) service_pool.cc:225] Handling call
0626 08:24:25.787539 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:25.900404 (+112865us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:26.840331 (+939927us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54005442d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:26.840505 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257907968}
I20250626 08:24:26.815524 5338 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:26.750174 5393 raft_consensus.cc:2747] T 75222ddacb444f7ba27de92b1571ada8 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:26.850572 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:26.857272 1967 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:26.830099 5214 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.616s user 0.001s sys 0.035s
I20250626 08:24:26.736320 5437 raft_consensus.cc:513] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:26.867956 5444 log.cc:927] Time spent T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20250626 08:24:26.868234 5437 leader_election.cc:290] T 7a16389239e54c52bb619a62b1b0b91a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:26.869069 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2167) took 2206 ms. Trace:
I20250626 08:24:26.870119 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.870932 1919 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:26.827412 5331 raft_consensus.cc:2802] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:26.602620 5368 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 3 FOLLOWER]: Advancing to term 4
I20250626 08:24:26.835556 5333 raft_consensus.cc:491] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:26.841760 5380 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250626 08:24:26.874032 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:26.869436 1920 rpcz_store.cc:276] 0626 08:24:24.662994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.689302 (+ 26308us) service_pool.cc:225] Handling call
0626 08:24:24.702717 (+ 13415us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:24.960490 (+257773us) raft_consensus.cc:1532] Early marking committed up to index 2
0626 08:24:24.960525 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:24.961061 (+ 536us) log.cc:844] Serialized 26 byte log entry
0626 08:24:25.949483 (+988422us) raft_consensus.cc:1643] Marking committed up to 2
0626 08:24:25.949520 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.949582 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:26.868843 (+919261us) raft_consensus.cc:1683] finished
0626 08:24:26.868863 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:26.868984 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85355,"raft.queue_time_us":325254,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":1107570,"threads_started":2,"wal-append.queue_time_us":986024}
W20250626 08:24:26.856950 5322 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 2.929s user 0.044s sys 0.052s
I20250626 08:24:26.723075 5420 raft_consensus.cc:513] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } attrs { replace: true } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.883095 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.838748 1993 rpcz_store.cc:276] 0626 08:24:24.162155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.546210 (+384055us) service_pool.cc:225] Handling call
0626 08:24:25.640409 (+1094199us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:25.641162 (+ 753us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:26.837976 (+1196814us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540039c450. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907e49d 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:26.838167 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2515439872}
I20250626 08:24:26.833878 1849 rpcz_store.cc:276] 0626 08:24:23.979889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.980018 (+ 129us) service_pool.cc:225] Handling call
0626 08:24:24.002246 (+ 22228us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:24.170211 (+167965us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.170244 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:24.227317 (+ 57073us) log.cc:844] Serialized 78 byte log entry
0626 08:24:25.684762 (+1457445us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:25.706187 (+ 21425us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.706650 (+ 463us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.706966 (+ 316us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.707463 (+ 497us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.707573 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:26.768262 (+1060689us) raft_consensus.cc:1683] finished
0626 08:24:26.768293 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:26.768505 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118313,"prepare.queue_time_us":110444,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":1785,"raft.queue_time_us":133065,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7484,"spinlock_wait_cycles":46777856,"thread_start_us":1417537,"threads_started":1,"wal-append.queue_time_us":1816446}
I20250626 08:24:26.880383 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.856576 5289 raft_consensus.cc:3053] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 08:24:26.857955 5300 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Advancing to term 9
I20250626 08:24:26.891345 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:26.870031 5371 raft_consensus.cc:2747] T 9122458a1eae4f07b1a7f4ff63d6be19 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 7. Current term is 8.
I20250626 08:24:26.895098 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.895895 5289 raft_consensus.cc:738] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:26.896179 1921 raft_consensus.cc:2408] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20250626 08:24:26.899936 4916 raft_consensus.cc:3053] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 08:24:26.900453 4916 raft_consensus.cc:738] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:26.902912 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.903277 5355 leader_election.cc:290] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Requested vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:26.873067 5324 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:26.904099 5324 raft_consensus.cc:491] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:26.904479 5324 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:26.904824 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:26.885414 1920 raft_consensus.cc:2466] T 178e69a3d3834cae8db6684d32c1c4ce P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 11.
I20250626 08:24:26.886029 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e7d84adc07945fe8a3aa16f71b111e3" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.875758 5330 raft_consensus.cc:2802] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 9 FOLLOWER]: Leader election won for term 9
I20250626 08:24:26.873435 5331 raft_consensus.cc:491] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:26.909761 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:26.888594 1996 raft_consensus.cc:2408] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250626 08:24:26.889437 5420 leader_election.cc:290] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:26.882671 5322 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 3.487s user 0.045s sys 0.052s
I20250626 08:24:26.879527 5333 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:26.897683 1921 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:26.929317 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.932554 1919 raft_consensus.cc:2466] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:26.933753 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.934553 1919 raft_consensus.cc:2466] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
I20250626 08:24:26.875891 5328 leader_election.cc:290] T dc905d51bc4a450484091ffa39379396 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:26.929773 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:26.938750 1921 raft_consensus.cc:2391] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:26.882858 5380 raft_consensus.cc:491] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:26.944458 1926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2164) took 2739 ms. Trace:
I20250626 08:24:26.910135 1967 leader_election.cc:304] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:26.889497 1841 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1980) took 2695 ms. Trace:
I20250626 08:24:26.945005 1926 rpcz_store.cc:276] 0626 08:24:24.205083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.296556 (+ 91473us) service_pool.cc:225] Handling call
0626 08:24:24.325875 (+ 29319us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:24.687129 (+361254us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.694418 (+ 7289us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:24.713126 (+ 18708us) log.cc:844] Serialized 52 byte log entry
0626 08:24:25.492755 (+779629us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:25.504785 (+ 12030us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.508192 (+ 3407us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.508418 (+ 226us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.508600 (+ 182us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:26.620757 (+1112157us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b18002f02c0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1
0626 08:24:26.904981 (+284224us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b2400276ea8. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb5f77bf4 00007f4eb5f75cfb 00007f4eb6004ab9 00007f4eb6004745 00007f4eb6003ca0 00007f4eb8ceb228 00007f4eb8ceb988 00007f4eb8cc31c8 00007f4eb8cbe5aa 00007f4eb8cbd9f6
0626 08:24:26.944143 (+ 39162us) raft_consensus.cc:1683] finished
0626 08:24:26.944172 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:26.944362 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98890,"prepare.queue_time_us":1360976,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":11805,"raft.queue_time_us":263564,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":3196,"spinlock_wait_cycles":22021248,"thread_start_us":785576,"threads_started":2,"wal-append.queue_time_us":976703}
I20250626 08:24:26.763983 5343 raft_consensus.cc:2802] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:26.763803 5422 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:26.875487 5214 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20250626 08:24:26.907266 5002 leader_election.cc:290] T 8e262c3543404a9cb840169098735ea2 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:26.909475 5375 leader_election.cc:290] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:26.911669 5331 raft_consensus.cc:3058] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:26.901320 5289 consensus_queue.cc:260] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:26.945767 1841 rpcz_store.cc:276] 0626 08:24:24.194288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.011562 (+817274us) service_pool.cc:225] Handling call
0626 08:24:25.027267 (+ 15705us) raft_consensus.cc:1469] Updating replica for 3 ops
0626 08:24:25.050398 (+ 23131us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:25.050440 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0626 08:24:25.072582 (+ 22142us) log.cc:844] Serialized 78 byte log entry
0626 08:24:25.901769 (+829187us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:25.906412 (+ 4643us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.906793 (+ 381us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.906977 (+ 184us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.907070 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:26.889217 (+982147us) raft_consensus.cc:1683] finished
0626 08:24:26.889248 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:26.889427 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16817,"prepare.queue_time_us":4358,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":142673,"raft.run_cpu_time_us":475,"raft.run_wall_time_us":8483,"thread_start_us":795564,"threads_started":1,"wal-append.queue_time_us":1527866}
I20250626 08:24:26.930624 5330 raft_consensus.cc:695] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [term 9 LEADER]: Becoming Leader. State: Replica: 755d83ca0913406b816e95415d502055, State: Running, Role: LEADER
W20250626 08:24:26.967851 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 57789.4us
I20250626 08:24:26.952565 5380 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Advancing to term 10
I20250626 08:24:26.760425 5433 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:26.971416 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:26.965713 1918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 650) took 2811 ms. Trace:
I20250626 08:24:26.973368 1967 leader_election.cc:304] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:26.975849 2583 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.191s user 0.065s sys 0.101s
I20250626 08:24:26.974040 1919 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:26.976449 1967 leader_election.cc:304] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:26.976176 2583 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.191s user 0.065s sys 0.101s
I20250626 08:24:26.979017 1967 leader_election.cc:304] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:26.991048 1967 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
W20250626 08:24:26.978552 5007 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 1.224s user 0.060s sys 0.136s
I20250626 08:24:26.973937 1918 rpcz_store.cc:276] 0626 08:24:24.154161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.209340 (+ 55179us) service_pool.cc:225] Handling call
0626 08:24:24.209652 (+ 312us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:24.304497 (+ 94845us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.304541 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:24.349000 (+ 44459us) log.cc:844] Serialized 26 byte log entry
0626 08:24:25.028494 (+679494us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b18001c5ac0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:25.683246 (+654752us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:25.683310 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.683407 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:26.960948 (+1277541us) raft_consensus.cc:1683] finished
0626 08:24:26.960971 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:26.965625 (+ 4654us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140918,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"spinlock_wait_cycles":18335104,"thread_start_us":1214104,"threads_started":2,"wal-append.queue_time_us":2537404}
I20250626 08:24:26.973352 5414 raft_consensus.cc:2802] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:26.973659 5289 raft_consensus.cc:3058] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:27.012523 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:26.992937 5375 raft_consensus.cc:2802] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250626 08:24:26.993886 5352 leader_election.cc:290] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 11 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:26.990047 5323 raft_consensus.cc:2747] T 7e7d84adc07945fe8a3aa16f71b111e3 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:26.901413 4916 consensus_queue.cc:260] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:27.007453 5007 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 1.255s user 0.060s sys 0.136s
I20250626 08:24:27.003190 5420 raft_consensus.cc:2802] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250626 08:24:27.045570 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:27.043861 1918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:27.057752 1812 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.164s user 0.016s sys 0.025s
I20250626 08:24:26.993105 5227 consensus_queue.cc:1035] T 0914c29163894ada9540f1133a461648 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20250626 08:24:27.010196 5422 raft_consensus.cc:2802] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:27.032176 5414 raft_consensus.cc:491] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
W20250626 08:24:27.056988 1925 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.019s sys 0.014s
W20250626 08:24:27.065804 1812 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.172s user 0.017s sys 0.025s
W20250626 08:24:27.073431 1925 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s user 0.019s sys 0.015s
I20250626 08:24:27.058918 5420 raft_consensus.cc:491] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.043433 5375 raft_consensus.cc:491] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.060132 5342 leader_election.cc:290] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:27.042537 1814 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.115s user 0.012s sys 0.028s
I20250626 08:24:27.055572 5270 leader_election.cc:290] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.076279 5269 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.083204 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2bf9541d0634f9fab914afd74bb9505" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.095280 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:27.082561 1841 raft_consensus.cc:1273] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:27.096355 1843 raft_consensus.cc:2391] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 7: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:27.095402 1849 raft_consensus.cc:2466] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
I20250626 08:24:27.101975 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34656 (request call id 1850) took 2104 ms. Trace:
I20250626 08:24:27.068504 5422 raft_consensus.cc:491] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:27.073750 5414 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:27.125133 1843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:27.093815 1814 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.167s user 0.012s sys 0.029s
W20250626 08:24:27.121276 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.125988 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 1.1997e+06us
I20250626 08:24:27.126104 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.041409 2580 raft_consensus.cc:3058] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Advancing to term 11
I20250626 08:24:27.102504 1995 rpcz_store.cc:276] 0626 08:24:24.988905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.989103 (+ 198us) service_pool.cc:225] Handling call
0626 08:24:27.086028 (+2096925us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b5400280fd4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:27.093437 (+ 7409us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1488647424}
I20250626 08:24:27.125613 5422 raft_consensus.cc:3058] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:27.161603 1849 raft_consensus.cc:1238] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 5. Current term is 6. Ops: []
I20250626 08:24:27.086499 5420 raft_consensus.cc:3058] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:27.076947 4916 raft_consensus.cc:3058] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:27.165616 5389 raft_consensus.cc:491] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:27.169127 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "423807ecf2d54e869ffc86a272ebd10a" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.069339 5330 consensus_queue.cc:237] T c014cb521ac2424fa52882a7d676dbec P 755d83ca0913406b816e95415d502055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.117954 5269 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.095288 5375 raft_consensus.cc:3058] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 11 FOLLOWER]: Advancing to term 12
I20250626 08:24:27.160110 5431 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.162148 5270 raft_consensus.cc:491] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:27.191996 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:27.204025 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.175839 1968 thread.cc:640] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.074s user 0.020s sys 0.010s
I20250626 08:24:27.186762 5389 raft_consensus.cc:513] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.209108 5389 leader_election.cc:290] T 7846e33f4b874a229eb7250f99a6db61 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:27.210470 5446 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P 755d83ca0913406b816e95415d502055: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20250626 08:24:27.212072 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "178e69a3d3834cae8db6684d32c1c4ce" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:27.212352 1968 thread.cc:606] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.110s user 0.021s sys 0.011s
W20250626 08:24:27.214169 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 1.13005e+06us
I20250626 08:24:27.186220 5355 raft_consensus.cc:491] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.167057 5300 raft_consensus.cc:513] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.221581 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:27.222779 1849 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 9: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:27.202176 5433 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:27.245978 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.356956 5441 log.cc:927] Time spent T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.301s user 0.003s sys 0.004s
I20250626 08:24:27.197196 5431 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:27.373304 4662 consensus_meta.cc:287] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.516s user 0.008s sys 0.000s
I20250626 08:24:27.373023 5333 raft_consensus.cc:513] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.373793 5431 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.176744 1841 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "148557a00636474cad8673c1f02aae5e"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:27.374125 5352 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:27.374290 5269 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:27.257042 5448 log.cc:927] Time spent T 7846e33f4b874a229eb7250f99a6db61 P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20250626 08:24:27.224164 5355 raft_consensus.cc:513] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.225251 5368 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:27.278147 5449 log.cc:927] Time spent T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
W20250626 08:24:27.282591 1893 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.026s sys 0.041s
W20250626 08:24:27.391867 1893 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.026s sys 0.041s
W20250626 08:24:27.392553 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 805900us
W20250626 08:24:27.282213 2551 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.019s sys 0.047s
I20250626 08:24:27.345342 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:27.393784 2551 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.757s user 0.025s sys 0.054s
I20250626 08:24:27.394091 1842 raft_consensus.cc:2466] T 5549f2d4a32b4170ad59fc7489d1365f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
W20250626 08:24:27.245431 2550 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.023s sys 0.037s
W20250626 08:24:27.395010 2550 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.028s sys 0.050s
W20250626 08:24:27.345882 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.347949 1927 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.010s sys 0.014s
W20250626 08:24:27.395800 1927 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.694s user 0.010s sys 0.014s
W20250626 08:24:27.215781 1968 connection.cc:232] RPC call timeout handler was delayed by -1.0594463610000275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20250626 08:24:27.396838 1968 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.348420 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.013s sys 0.002s
I20250626 08:24:27.351605 2580 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 11.
W20250626 08:24:27.397292 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.014s sys 0.002s
W20250626 08:24:27.397611 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 671152us
W20250626 08:24:27.395370 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 759840us
W20250626 08:24:27.352543 5442 log.cc:927] Time spent T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.198s user 0.006s sys 0.011s
I20250626 08:24:27.201331 5270 raft_consensus.cc:513] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:27.366251 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:27.370043 5324 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.418777 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 756) took 1638 ms. Trace:
I20250626 08:24:27.419097 1919 raft_consensus.cc:1238] T a71e0323b96e4a47b1ba2e0a3d72840f P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 6. Current term is 7. Ops: []
I20250626 08:24:27.419724 1841 raft_consensus.cc:604] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:27.375468 1925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2174) took 1597 ms. Trace:
I20250626 08:24:27.420128 1841 raft_consensus.cc:612] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250626 08:24:27.394181 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 757153us
I20250626 08:24:27.398413 1890 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:27.398942 2550 leader_election.cc:304] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
W20250626 08:24:27.237959 1816 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 1.348s user 0.044s sys 0.059s
W20250626 08:24:27.425287 1816 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 1.536s user 0.057s sys 0.068s
I20250626 08:24:27.395836 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "766debd72f3f4feaa9e81cd3690afdc1" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:27.398350 1965 leader_election.cc:304] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:27.426844 1965 leader_election.cc:304] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:27.426957 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d1eb5f502d8430b9109e246276cc44e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.428021 5227 raft_consensus.cc:2747] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:27.374763 5352 raft_consensus.cc:513] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.428690 5227 raft_consensus.cc:2802] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:27.429044 5227 raft_consensus.cc:491] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.429374 5227 raft_consensus.cc:3058] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:27.419355 2583 rpcz_store.cc:276] 0626 08:24:25.780439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.780567 (+ 128us) service_pool.cc:225] Handling call
0626 08:24:25.783555 (+ 2988us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:25.783918 (+ 363us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:25.783949 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:25.784477 (+ 528us) log.cc:844] Serialized 26 byte log entry
0626 08:24:26.980025 (+1195548us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:26.980083 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:26.980178 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:27.418543 (+438365us) raft_consensus.cc:1683] finished
0626 08:24:27.418565 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.418719 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":357,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":1195126,"threads_started":1,"wal-append.queue_time_us":1578697}
I20250626 08:24:27.420239 1925 rpcz_store.cc:276] 0626 08:24:25.778301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.778431 (+ 130us) service_pool.cc:225] Handling call
0626 08:24:25.784676 (+ 6245us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:25.785039 (+ 363us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:25.785073 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:25.854446 (+ 69373us) log.cc:844] Serialized 26 byte log entry
0626 08:24:27.074248 (+1219802us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:27.119474 (+ 45226us) log.cc:844] Serialized 14 byte log entry
0626 08:24:27.120420 (+ 946us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:27.120519 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:27.375256 (+254737us) raft_consensus.cc:1683] finished
0626 08:24:27.375279 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.375410 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9526,"prepare.queue_time_us":56152,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3785,"raft.queue_time_us":907,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":54283,"thread_start_us":1109412,"threads_started":1,"wal-append.queue_time_us":1425181}
I20250626 08:24:27.371218 5380 raft_consensus.cc:513] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.402074 5323 raft_consensus.cc:491] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:27.406626 5270 leader_election.cc:290] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
W20250626 08:24:27.414994 5447 log.cc:927] Time spent T f9b805acc8ae4c3abfe6d6ee40fbf456 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.362s user 0.002s sys 0.000s
I20250626 08:24:27.415551 1848 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60136 (request call id 1978) took 3222 ms. Trace:
I20250626 08:24:27.416611 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331d5bdb3fda41489445725e41b94bd1" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:27.418846 1928 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58462 (request call id 655) took 1465 ms (client timeout 1786 ms). Trace:
W20250626 08:24:27.439106 1928 rpcz_store.cc:269] 0626 08:24:25.953824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.955955 (+ 2131us) service_pool.cc:225] Handling call
0626 08:24:27.418655 (+1462700us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b540030f554. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:27.418798 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1961907456}
I20250626 08:24:27.438730 1848 rpcz_store.cc:276] 0626 08:24:24.193639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.928181 (+734542us) service_pool.cc:225] Handling call
0626 08:24:24.931530 (+ 3349us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:24.994071 (+ 62541us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:24.994477 (+ 406us) log.cc:844] Serialized 14 byte log entry
0626 08:24:26.850791 (+1856314us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:26.851392 (+ 601us) log.cc:844] Serialized 52 byte log entry
0626 08:24:26.851544 (+ 152us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:27.311255 (+459711us) log.cc:844] Serialized 14 byte log entry
0626 08:24:27.311600 (+ 345us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:27.311720 (+ 120us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:27.415322 (+103602us) raft_consensus.cc:1683] finished
0626 08:24:27.415341 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.415492 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54057,"prepare.queue_time_us":804,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":9202,"raft.queue_time_us":148653,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":23486,"thread_start_us":2298275,"threads_started":2,"wal-append.queue_time_us":2077271}
I20250626 08:24:27.418820 1847 raft_consensus.cc:1273] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:27.401573 1893 leader_election.cc:400] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
W20250626 08:24:27.418094 1968 leader_election.cc:336] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311): Timed out: RequestConsensusVote RPC to 127.0.171.193:43311 timed out after 2.007s (SENT)
I20250626 08:24:27.443518 1893 leader_election.cc:403] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20250626 08:24:27.425592 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 1.53619e+06us
I20250626 08:24:27.436507 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.437372 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.373705 4662 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.447602 1816 leader_election.cc:304] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:27.449261 1816 leader_election.cc:304] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:27.457443 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.446808 2581 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:27.461886 2579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.463007 1842 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250626 08:24:27.463263 2585 raft_consensus.cc:1238] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Rejecting Update request from peer 755d83ca0913406b816e95415d502055 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20250626 08:24:27.463855 2002 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41790 (request call id 512) took 1346 ms. Trace:
I20250626 08:24:27.465996 2002 rpcz_store.cc:276] 0626 08:24:26.117344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:26.117463 (+ 119us) service_pool.cc:225] Handling call
0626 08:24:27.416428 (+1298965us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e2d62 00007f4ebedd08f9 00007f4ebdf5bac0 00007f4ebe0227bb 00007f4ebe022841 00007f4ebdfd15e0 00007f4ebdfd0ec4 00007f4eb766ccc4 00007f4eb766cc52
0626 08:24:27.463801 (+ 47373us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5177216}
I20250626 08:24:27.457021 2550 leader_election.cc:304] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:27.467052 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.466917 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.467617 1928 raft_consensus.cc:2466] T 5f31173b902a46949ba221ed157ca482 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 6.
I20250626 08:24:27.470640 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.471510 2580 raft_consensus.cc:2466] T b498813b859f40cd95ed6adf6a2d8918 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:27.473098 1890 leader_election.cc:304] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:27.438711 5351 raft_consensus.cc:491] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.436184 5352 leader_election.cc:290] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.483675 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.484333 1928 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:27.402935 5414 raft_consensus.cc:513] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.465047 5270 raft_consensus.cc:2802] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 08:24:27.492306 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8539685c775453b8aaf4921fa90fa63" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.461848 5329 raft_consensus.cc:2747] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250626 08:24:27.499583 2587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36144 (request call id 750) took 4164 ms. Trace:
I20250626 08:24:27.484620 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.494661 1814 leader_election.cc:304] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:27.487617 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.502787 1814 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
W20250626 08:24:27.509758 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:27.502158 1925 raft_consensus.cc:2466] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:27.516012 1967 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:27.517640 1816 leader_election.cc:304] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:27.466699 5459 raft_consensus.cc:2802] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:27.521970 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:27.498243 5291 consensus_meta.cc:287] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6: Time spent flushing consensus metadata: real 0.637s user 0.008s sys 0.004s
I20250626 08:24:27.442276 5323 raft_consensus.cc:513] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.501726 2587 rpcz_store.cc:276] 0626 08:24:23.335158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:23.350559 (+ 15401us) service_pool.cc:225] Handling call
0626 08:24:23.391743 (+ 41184us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:25.582511 (+2190768us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b1800304240. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb8ccb2ec 00007f4eb610324c 00007f4ebed309b9 00007f4eb611f79b 00007f4eb611da3f 00007f4eb611d555 00007f4eb9076c50 00007f4eb90777eb 00007f4eb908222b
0626 08:24:25.582692 (+ 181us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:25.582734 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:25.590476 (+ 7742us) log.cc:844] Serialized 26 byte log entry
0626 08:24:26.756224 (+1165748us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:26.756283 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:26.756390 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:27.499204 (+742814us) raft_consensus.cc:1683] finished
0626 08:24:27.499233 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.499505 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20606,"raft.queue_time_us":219589,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":29946,"spinlock_wait_cycles":2481408,"thread_start_us":1446018,"threads_started":2,"wal-append.queue_time_us":1477067}
I20250626 08:24:27.531843 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:27.532445 1925 raft_consensus.cc:3058] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:27.521255 5271 consensus_queue.cc:1035] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20250626 08:24:27.523190 5270 raft_consensus.cc:695] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:27.518362 5411 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:27.523512 5300 leader_election.cc:290] T 70ce63b4769e4ca8980ddb20229389ef P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.487231 5007 raft_consensus.cc:2802] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250626 08:24:27.526870 5459 raft_consensus.cc:491] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.487012 5352 raft_consensus.cc:2747] T 5697952652a3463cbe2ee01e18348697 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311) with higher term. Message: Invalid argument: T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
I20250626 08:24:27.491035 5289 consensus_queue.cc:1035] T 178e69a3d3834cae8db6684d32c1c4ce P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20250626 08:24:27.484534 5457 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 FOLLOWER]: Leader election won for term 11
I20250626 08:24:27.490219 5351 raft_consensus.cc:513] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.514971 5454 raft_consensus.cc:2802] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:27.487162 5452 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20250626 08:24:27.498031 5333 leader_election.cc:290] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.583353 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.533376 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:27.542155 1813 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:27.584234 1842 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 7.
W20250626 08:24:27.549441 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34646 (request call id 2323) took 1679 ms (client timeout 2016 ms). Trace:
W20250626 08:24:27.585021 1996 rpcz_store.cc:269] 0626 08:24:25.869810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:25.885857 (+ 16047us) service_pool.cc:225] Handling call
0626 08:24:27.549230 (+1663373us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54001306d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb9082eda 00007f4ebdfd1057 00007f4eb766ccc4 00007f4eb766cc52 00007f4eb766cb81 00007f4eb766cafc 00007f4eb766bdb2 00007f4eb7481dcc 00007f4eb7481206 00007f4eb7480baf
0626 08:24:27.549380 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2112931072}
I20250626 08:24:27.550118 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.559099 2000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41790 (request call id 507) took 1443 ms. Trace:
W20250626 08:24:27.565865 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:27.573760 1845 raft_consensus.cc:2466] T 766debd72f3f4feaa9e81cd3690afdc1 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 4.
I20250626 08:24:27.569531 2584 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.544139 1919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.572621 2581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.548197 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736124bc239f4257b44dfc38e3acc17f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:27.543856 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.547976 2582 raft_consensus.cc:1273] T 3d1eb5f502d8430b9109e246276cc44e P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250626 08:24:27.547490 2579 raft_consensus.cc:1238] T cd7130b92c4840deaca578e523eb4a09 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 8. Current term is 9. Ops: []
I20250626 08:24:27.590911 1841 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:27.562983 1812 connection.cc:232] RPC call timeout handler was delayed by -2.7497175319999769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2738s for a response
I20250626 08:24:27.568631 1925 raft_consensus.cc:2466] T 856b2d0e8d2948bd9bfc33345dcd4886 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 6.
W20250626 08:24:27.592104 1812 connection.cc:232] RPC call timeout handler was delayed by -1.0283716459998686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
I20250626 08:24:27.592445 1925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:27.574877 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 50956.4us
I20250626 08:24:27.593319 1925 raft_consensus.cc:2408] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250626 08:24:27.594262 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5329c15e45ba48438209a93439414108" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.594939 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26cfef3699a141f8b7686ae2b75a3beb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.595152 1845 raft_consensus.cc:2466] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 10.
I20250626 08:24:27.599148 2581 raft_consensus.cc:2391] T 3e928895bcfa4d00823f3f4b50eebf86 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 7: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
I20250626 08:24:27.600241 1812 leader_election.cc:304] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:27.586735 2000 rpcz_store.cc:276] 0626 08:24:26.115931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:26.116077 (+ 146us) service_pool.cc:225] Handling call
0626 08:24:27.553100 (+1437023us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b58002d0c98. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb60e2eb1 00007f4eb60e322e 0000000000356d09 00007f4ebdf5bdc0 00007f4ebe0227d4 00007f4ebe022841 00007f4ebdfe5860 00007f4ebdfd0c29 00007f4ebdfd06d2 00007f4eb766a744
0626 08:24:27.553266 (+ 166us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:27.558154 (+ 4888us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:27.558185 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:27.558228 (+ 43us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:27.558258 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:27.558895 (+ 637us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b5400270454. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:27.558940 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.559044 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":339945600}
I20250626 08:24:27.535284 5343 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.538064 5269 raft_consensus.cc:2802] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:27.602249 2586 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.464406 5422 raft_consensus.cc:513] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:27.558883 5439 log.cc:927] Time spent T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.793s user 0.003s sys 0.000s
I20250626 08:24:27.604079 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.602241 1846 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b805acc8ae4c3abfe6d6ee40fbf456" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:27.603605 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 57474.8us
I20250626 08:24:27.606724 1848 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.633881 1928 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:27.609545 5289 raft_consensus.cc:2802] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 FOLLOWER]: Leader election won for term 10
I20250626 08:24:27.555650 5324 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:27.559123 5323 leader_election.cc:290] T 171f89014c824d17a510a3cc17b3af0d P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.533814 5271 consensus_queue.cc:1230] T d9f49ece70144b92b201204afc3042ed P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer 755d83ca0913406b816e95415d502055 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250626 08:24:27.550606 5352 raft_consensus.cc:2802] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:27.603258 5286 raft_consensus.cc:491] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:27.565768 4956 consensus_queue.cc:1035] T dc905d51bc4a450484091ffa39379396 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20250626 08:24:27.610329 1850 raft_consensus.cc:1273] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250626 08:24:27.566082 5214 consensus_queue.cc:1035] T 5549f2d4a32b4170ad59fc7489d1365f P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20250626 08:24:27.610535 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd7130b92c4840deaca578e523eb4a09" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:27.605386 5269 raft_consensus.cc:491] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.553747 5380 leader_election.cc:290] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.547225 5457 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 11 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:27.552524 5465 raft_consensus.cc:491] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:27.555019 5445 raft_consensus.cc:2802] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:27.747079 5445 raft_consensus.cc:491] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.747488 5445 raft_consensus.cc:3058] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:27.535784 5270 consensus_queue.cc:237] T 4a65a55472e64be4b9ae56426278c3c1 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.536073 5411 raft_consensus.cc:491] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:27.752697 5411 raft_consensus.cc:3058] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:27.612342 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:27.758472 1849 raft_consensus.cc:3058] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Advancing to term 9
W20250626 08:24:27.612975 5331 consensus_meta.cc:287] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.641s user 0.011s sys 0.000s
I20250626 08:24:27.762296 5331 raft_consensus.cc:513] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.613559 2587 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.537678 5459 raft_consensus.cc:3058] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:27.537317 5007 raft_consensus.cc:491] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.614189 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.781384 1842 raft_consensus.cc:2391] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 6: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:27.614842 2581 raft_consensus.cc:1273] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250626 08:24:27.674474 5289 raft_consensus.cc:695] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 10 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:27.616170 5414 leader_election.cc:290] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:27.616587 5227 raft_consensus.cc:513] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.602409 5343 raft_consensus.cc:3058] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:27.619684 1812 leader_election.cc:304] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:27.623288 2583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148557a00636474cad8673c1f02aae5e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" is_pre_election: true
I20250626 08:24:27.625011 5317 consensus_queue.cc:1035] T 9122458a1eae4f07b1a7f4ff63d6be19 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250626 08:24:27.625495 2550 leader_election.cc:304] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:27.567523 5412 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.627089 1925 raft_consensus.cc:1273] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Refusing update from remote peer 755d83ca0913406b816e95415d502055: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250626 08:24:27.570032 5291 raft_consensus.cc:513] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.629498 1847 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250626 08:24:27.598392 5458 consensus_queue.cc:1035] T 856b2d0e8d2948bd9bfc33345dcd4886 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20250626 08:24:27.607411 1816 leader_election.cc:304] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:27.634902 1928 raft_consensus.cc:2466] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:27.552955 5454 raft_consensus.cc:491] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.669327 1845 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.534873 4662 leader_election.cc:290] T cef3ff80065343cdbaf58ef8b1bbedf9 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.556416 5355 leader_election.cc:290] T bcf570bd4fbe46dc9453a7319f45a376 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.674080 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58462 (request call id 668) took 1135 ms. Trace:
I20250626 08:24:27.702335 2580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5329c15e45ba48438209a93439414108"
dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:45504
I20250626 08:24:27.742518 5269 raft_consensus.cc:3058] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:27.743739 5422 leader_election.cc:290] T fec12ab89e754fea8f2b9e353bba3dee P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
W20250626 08:24:27.720100 5420 consensus_meta.cc:287] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.535s user 0.008s sys 0.002s
I20250626 08:24:27.614951 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ba4a836fec402199d1aecf04c04b62" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:27.782303 2587 raft_consensus.cc:2391] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate be56082cacb54e9aa8f1a9ee91e70d76 in this term.
W20250626 08:24:27.765247 1918 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.005s sys 0.026s
W20250626 08:24:27.783071 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 389438us
I20250626 08:24:27.745378 5286 raft_consensus.cc:513] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.819741 5435 raft_consensus.cc:3058] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:27.835069 1918 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.006s sys 0.026s
W20250626 08:24:27.798489 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 178876us
W20250626 08:24:27.799322 1891 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:27.801975 2550 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 176539us
I20250626 08:24:27.837128 1814 leader_election.cc:304] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:27.786190 5007 raft_consensus.cc:3058] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Advancing to term 6
I20250626 08:24:27.825040 2580 raft_consensus.cc:604] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 11 FOLLOWER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:27.841120 2580 raft_consensus.cc:612] T 5329c15e45ba48438209a93439414108 P be56082cacb54e9aa8f1a9ee91e70d76 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:27.825210 1924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2170) took 3114 ms. Trace:
W20250626 08:24:27.822851 1816 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 215506us
I20250626 08:24:27.783618 1849 raft_consensus.cc:2408] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20250626 08:24:27.828459 1843 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:27.820639 5375 consensus_meta.cc:287] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.000s
I20250626 08:24:27.833491 1893 leader_election.cc:304] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:27.817670 5351 leader_election.cc:290] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.841881 1924 rpcz_store.cc:276] 0626 08:24:24.710849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.898966 (+188117us) service_pool.cc:225] Handling call
0626 08:24:24.909394 (+ 10428us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:24.944707 (+ 35313us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.944744 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:24.945468 (+ 724us) log.cc:844] Serialized 52 byte log entry
0626 08:24:25.641046 (+695578us) raft_consensus.cc:1643] Marking committed up to 6
0626 08:24:25.641084 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:26.059905 (+418821us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b54002af3d4. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb9003481 00007f4eb907f9f1 00007f4eb907e42e 00007f4ebdfd0719 00007f4eb766a744 00007f4eb766a6d2 00007f4eb766a601 00007f4eb766a57c 00007f4eb7669832 00007f4eb7481dcc 00007f4eb7481206
0626 08:24:26.059987 (+ 82us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:27.558327 (+1498340us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b440002304c. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb6031ff5 00007f4eb6032a12 00007f4eb8ccefc1 00007f4eb8cca8ea 00007f4eb8cbe6ae 00007f4eb8cbd9f6 00007f4eb8cbd618 00007f4eb8cc7051 00007f4eb8cc7009 00007f4eb8cc6f99
0626 08:24:27.824728 (+266401us) raft_consensus.cc:1683] finished
0626 08:24:27.824776 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.825154 (+ 378us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41660,"raft.queue_time_us":127358,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":1276,"spinlock_wait_cycles":49152256,"thread_start_us":637775,"threads_started":1,"wal-append.queue_time_us":1776790}
I20250626 08:24:27.834614 1928 raft_consensus.cc:1273] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20250626 08:24:27.848541 1928 pending_rounds.cc:85] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting all ops after (but not including) 6
I20250626 08:24:27.848834 1928 pending_rounds.cc:107] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20250626 08:24:27.849107 1928 raft_consensus.cc:2887] T 423807ecf2d54e869ffc86a272ebd10a P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:27.785336 5465 raft_consensus.cc:513] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.835315 1841 raft_consensus.cc:2466] T 94ba4a836fec402199d1aecf04c04b62 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 5.
I20250626 08:24:27.674489 5352 raft_consensus.cc:491] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 548099d81e7f46ac9b2f03d9e5e678a6)
I20250626 08:24:27.823858 5454 raft_consensus.cc:3058] T 5f31173b902a46949ba221ed157ca482 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Advancing to term 7
I20250626 08:24:27.841373 1812 leader_election.cc:304] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:27.823741 1927 rpcz_store.cc:276] 0626 08:24:26.538612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:26.646464 (+107852us) service_pool.cc:225] Handling call
0626 08:24:26.646751 (+ 287us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:26.646975 (+ 224us) raft_consensus.cc:1532] Early marking committed up to index 8
0626 08:24:27.396270 (+749295us) log.cc:844] Serialized 14 byte log entry
0626 08:24:27.673568 (+277298us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:27.673638 (+ 70us) raft_consensus.cc:1643] Marking committed up to 8
0626 08:24:27.673695 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:27.673787 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.674007 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":102455,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":74416,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":971205,"threads_started":2}
I20250626 08:24:27.827095 5002 raft_consensus.cc:2802] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:27.847965 5271 raft_consensus.cc:2802] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250626 08:24:27.854552 1924 raft_consensus.cc:1238] T 0914c29163894ada9540f1133a461648 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250626 08:24:27.860082 1893 leader_election.cc:304] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:27.868650 5271 raft_consensus.cc:491] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader be56082cacb54e9aa8f1a9ee91e70d76)
I20250626 08:24:27.871475 5271 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:27.871659 1924 raft_consensus.cc:1273] T c8539685c775453b8aaf4921fa90fa63 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Refusing update from remote peer 548099d81e7f46ac9b2f03d9e5e678a6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:27.873689 1893 leader_election.cc:304] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 755d83ca0913406b816e95415d502055
I20250626 08:24:27.865272 5002 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.876494 1893 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:27.876686 5002 raft_consensus.cc:3058] T cef3ff80065343cdbaf58ef8b1bbedf9 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:27.878859 5271 raft_consensus.cc:513] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.868647 5355 consensus_queue.cc:1046] T 4a65a55472e64be4b9ae56426278c3c1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250626 08:24:27.880990 5271 leader_election.cc:290] T 5e2fc654780843abbda82ba8f5df515e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.895458 5227 leader_election.cc:290] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:27.898995 5291 leader_election.cc:290] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.854727 5412 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.046s
I20250626 08:24:27.900730 5331 leader_election.cc:290] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 8 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:27.853653 5289 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:27.846949 5119 raft_consensus.cc:2747] T 70ce63b4769e4ca8980ddb20229389ef P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:27.883056 1923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2165) took 3221 ms. Trace:
I20250626 08:24:27.914179 1923 rpcz_store.cc:276] 0626 08:24:24.662145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:24.667394 (+ 5249us) service_pool.cc:225] Handling call
0626 08:24:24.691073 (+ 23679us) raft_consensus.cc:1469] Updating replica for 1 ops
0626 08:24:24.691437 (+ 364us) raft_consensus.cc:1532] Early marking committed up to index 0
0626 08:24:24.691465 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0626 08:24:24.711936 (+ 20471us) log.cc:844] Serialized 26 byte log entry
0626 08:24:25.075558 (+363622us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:25.094074 (+ 18516us) log.cc:844] Serialized 14 byte log entry
0626 08:24:25.094290 (+ 216us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:25.094359 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:25.373474 (+279115us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b18001256e0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb60ee3e6 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1
0626 08:24:25.397150 (+ 23676us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b18001256e0. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4ebedb7af1 00007f4ebedc4441 00007f4eb61204e2 00007f4eb61228b1 00007f4eb6122869 00007f4eb61227f9 00007f4eb61227c1 00007f4eb6121abd 00007f4ebf17b124 00007f4ebf17af59 00007f4eb6103636
0626 08:24:27.874769 (+2477619us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b2400276ea8. stack: 00007f4eb60ee483 00007f4eb60ee380 00007f4eb5a47574 00007f4ebf53b5a4 00007f4ebf53b539 00007f4eb5f639a1 00007f4eb5f77bf4 00007f4eb5f75cfb 00007f4eb6004ab9 00007f4eb6004745 00007f4eb6003ca0 00007f4eb8ceb228 00007f4eb8ceb988 00007f4eb8cc31c8 00007f4eb8cbe5aa 00007f4eb8cbd9f6
0626 08:24:27.882856 (+ 8087us) raft_consensus.cc:1683] finished
0626 08:24:27.882879 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:27.883000 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":20617,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":1294,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5584,"spinlock_wait_cycles":31863680,"thread_start_us":208275,"threads_started":1,"wal-append.queue_time_us":797681}
W20250626 08:24:27.908092 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 66792.2us
I20250626 08:24:27.917740 1812 leader_election.cc:304] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:27.907575 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 45573.3us
I20250626 08:24:27.919119 1812 leader_election.cc:304] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:27.835906 5420 raft_consensus.cc:513] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.882606 5339 raft_consensus.cc:491] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.854128 5380 raft_consensus.cc:2762] T c014cb521ac2424fa52882a7d676dbec P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20250626 08:24:27.856959 5352 raft_consensus.cc:3058] T b498813b859f40cd95ed6adf6a2d8918 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 6 FOLLOWER]: Advancing to term 7
W20250626 08:24:27.890848 5438 log.cc:927] Time spent T 8353d62d68914958b13bb87541c357f8 P 548099d81e7f46ac9b2f03d9e5e678a6: Append to log took a long time: real 0.778s user 0.005s sys 0.001s
W20250626 08:24:27.890386 1893 connection.cc:232] RPC call timeout handler was delayed by -3.5884277059999476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20250626 08:24:27.951701 1893 connection.cc:232] RPC call timeout handler was delayed by -3.5488277059998836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
W20250626 08:24:27.952209 1893 connection.cc:232] RPC call timeout handler was delayed by -3.5461277059998793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20250626 08:24:27.952702 1893 connection.cc:232] RPC call timeout handler was delayed by -3.3373277059999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
W20250626 08:24:27.953217 1893 connection.cc:232] RPC call timeout handler was delayed by -3.3157277059999615s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20250626 08:24:27.953660 1893 connection.cc:232] RPC call timeout handler was delayed by -3.2689277059998858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20250626 08:24:27.954142 1893 connection.cc:232] RPC call timeout handler was delayed by -3.1771277060001921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
W20250626 08:24:27.954591 1893 connection.cc:232] RPC call timeout handler was delayed by -3.0763277060000291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20250626 08:24:27.955047 1893 connection.cc:232] RPC call timeout handler was delayed by -3.0331277059999593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2235s for a response
W20250626 08:24:27.955482 1893 connection.cc:232] RPC call timeout handler was delayed by -2.9062277060002089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2376s for a response
W20250626 08:24:27.956055 1893 connection.cc:232] RPC call timeout handler was delayed by -2.3383277060002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3007s for a response
I20250626 08:24:27.903411 5465 leader_election.cc:290] T edeafceae4424506a7bffa7a9fbac201 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:28.033113 1999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
W20250626 08:24:27.958262 1920 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.035s sys 0.059s
W20250626 08:24:28.034420 1920 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.035s sys 0.059s
I20250626 08:24:27.881835 5434 raft_consensus.cc:491] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:27.884790 5368 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 4 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:27.927277 5418 raft_consensus.cc:2747] T ac6c556e015045ac949f4ee3d2048b56 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250626 08:24:27.854172 5375 raft_consensus.cc:513] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.935972 5396 raft_consensus.cc:2747] T 26cfef3699a141f8b7686ae2b75a3beb P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:27.936547 5431 raft_consensus.cc:2747] T 736124bc239f4257b44dfc38e3acc17f P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250626 08:24:27.929854 5271 raft_consensus.cc:491] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:27.934544 5329 consensus_queue.cc:1035] T 3d1eb5f502d8430b9109e246276cc44e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20250626 08:24:27.987143 1921 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.004s sys 0.016s
W20250626 08:24:27.988021 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 514963us
I20250626 08:24:27.988739 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
W20250626 08:24:28.043891 1921 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.005s sys 0.017s
I20250626 08:24:28.044229 1923 raft_consensus.cc:3058] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 3 FOLLOWER]: Advancing to term 5
I20250626 08:24:27.929065 5412 consensus_queue.cc:1035] T 423807ecf2d54e869ffc86a272ebd10a P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20250626 08:24:28.044608 1921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 2058) took 1176 ms. Trace:
W20250626 08:24:27.992731 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 476829us
I20250626 08:24:27.936985 5453 consensus_queue.cc:1046] T cd7130b92c4840deaca578e523eb4a09 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20250626 08:24:28.007124 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e928895bcfa4d00823f3f4b50eebf86" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.046591 1997 raft_consensus.cc:2466] T 3e928895bcfa4d00823f3f4b50eebf86 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:28.011821 5331 raft_consensus.cc:491] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:28.012123 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70ce63b4769e4ca8980ddb20229389ef" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:28.012341 2000 raft_consensus.cc:1273] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Refusing update from remote peer be56082cacb54e9aa8f1a9ee91e70d76: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20250626 08:24:28.047907 2000 pending_rounds.cc:85] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 6
I20250626 08:24:28.048236 2000 pending_rounds.cc:107] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250626 08:24:28.048460 2000 raft_consensus.cc:2887] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:28.048492 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f31173b902a46949ba221ed157ca482" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:27.985707 5380 raft_consensus.cc:2802] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250626 08:24:28.013763 2002 raft_consensus.cc:1238] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250626 08:24:27.853840 5457 consensus_queue.cc:237] T 5329c15e45ba48438209a93439414108 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.030135 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e6a8a739b954cc082a290e6c529a271" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.009984 1741 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:32876 (request call id 297) took 15615 ms (client timeout 15000 ms). Trace:
I20250626 08:24:27.947036 5395 consensus_queue.cc:1035] T c8539685c775453b8aaf4921fa90fa63 P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20250626 08:24:27.972852 5411 raft_consensus.cc:513] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:27.950567 5339 raft_consensus.cc:513] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
W20250626 08:24:28.036438 1968 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 471912us
I20250626 08:24:28.059252 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.040063 2551 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.193:43311, user_credentials={real_user=slave}} blocked reactor thread for 353162us
I20250626 08:24:28.045264 2580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84dfea36e4274e44b9a893967d69fd7e" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "be56082cacb54e9aa8f1a9ee91e70d76"
I20250626 08:24:28.060359 1993 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:28.060446 2580 raft_consensus.cc:3058] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Advancing to term 10
W20250626 08:24:28.046882 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:28.030606 1893 connection.cc:232] RPC call timeout handler was delayed by -2.2960277060001317s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3054s for a response
W20250626 08:24:28.054329 1892 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:28.061830 1890 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
W20250626 08:24:28.062160 1893 connection.cc:232] RPC call timeout handler was delayed by -2.1988277059999746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31620000000000004s for a response
W20250626 08:24:28.062577 1893 connection.cc:232] RPC call timeout handler was delayed by -1.1791277060001448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4295s for a response
W20250626 08:24:28.062944 1893 connection.cc:232] RPC call timeout handler was delayed by -1.106158349999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31410000000000005s for a response
I20250626 08:24:28.064438 1893 leader_election.cc:304] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055
I20250626 08:24:28.045009 1921 rpcz_store.cc:276] 0626 08:24:26.868175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:26.939478 (+ 71303us) service_pool.cc:225] Handling call
0626 08:24:26.939869 (+ 391us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:26.940121 (+ 252us) raft_consensus.cc:1532] Early marking committed up to index 7
0626 08:24:26.982769 (+ 42648us) log.cc:844] Serialized 14 byte log entry
0626 08:24:28.044277 (+1061508us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:28.044330 (+ 53us) raft_consensus.cc:1643] Marking committed up to 7
0626 08:24:28.044374 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:28.044435 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:28.044551 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53495,"prepare.queue_time_us":102251,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":89698,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":1030704,"threads_started":2,"wal-append.queue_time_us":551516}
I20250626 08:24:28.052639 5380 raft_consensus.cc:491] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 08:24:28.071784 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.072710 1921 raft_consensus.cc:3058] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:28.078819 2000 raft_consensus.cc:1238] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 7. Current term is 8. Ops: []
I20250626 08:24:28.068290 1968 leader_election.cc:304] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7
W20250626 08:24:28.084146 1967 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250626 08:24:28.063951 5465 consensus_queue.cc:1035] T 5697952652a3463cbe2ee01e18348697 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
W20250626 08:24:28.098016 5464 log.cc:927] Time spent T 5549f2d4a32b4170ad59fc7489d1365f P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250626 08:24:28.100610 5007 raft_consensus.cc:513] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:28.049752 1741 rpcz_store.cc:269] 0626 08:24:12.410197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:12.424747 (+ 14550us) service_pool.cc:225] Handling call
0626 08:24:27.998086 (+15573339us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0626 08:24:22.475121 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0626 08:24:22.475765 (+ 644us) write_op.cc:432] Acquiring schema lock in shared mode
0626 08:24:22.475818 (+ 53us) write_op.cc:435] Acquired schema lock
0626 08:24:22.475837 (+ 19us) tablet.cc:660] Decoding operations
0626 08:24:22.476951 (+ 1114us) write_op.cc:620] Acquiring the partition lock for write op
0626 08:24:22.477033 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0626 08:24:22.477062 (+ 29us) tablet.cc:683] Acquiring locks for 10 operations
0626 08:24:22.480033 (+ 2971us) tablet.cc:699] Row locks acquired
0626 08:24:22.480053 (+ 20us) write_op.cc:260] PREPARE: finished
0626 08:24:22.480235 (+ 182us) write_op.cc:270] Start()
0626 08:24:22.480457 (+ 222us) write_op.cc:276] Timestamp: P: 1750926262480167 usec, L: 0
0626 08:24:22.480567 (+ 110us) op_driver.cc:348] REPLICATION: starting
0626 08:24:22.481339 (+ 772us) log.cc:844] Serialized 4014 byte log entry
0626 08:24:23.305496 (+824157us) op_driver.cc:464] REPLICATION: finished
0626 08:24:23.674970 (+369474us) write_op.cc:301] APPLY: starting
0626 08:24:23.681144 (+ 6174us) tablet.cc:1365] starting BulkCheckPresence
0626 08:24:23.749575 (+ 68431us) tablet.cc:1368] finished BulkCheckPresence
0626 08:24:23.851495 (+101920us) tablet.cc:1370] starting ApplyRowOperation cycle
0626 08:24:23.998047 (+146552us) tablet.cc:1381] finished ApplyRowOperation cycle
0626 08:24:23.998343 (+ 296us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0626 08:24:23.998404 (+ 61us) write_op.cc:312] APPLY: finished
0626 08:24:23.998778 (+ 374us) log.cc:844] Serialized 77 byte log entry
0626 08:24:23.998967 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:23.999119 (+ 152us) write_op.cc:454] Released schema lock
0626 08:24:23.999184 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0626 08:24:24.031103 (+ 31919us) write_op.cc:489] Releasing partition, row and schema locks
0626 08:24:24.031133 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8853,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":222117,"apply.run_cpu_time_us":8559,"apply.run_wall_time_us":426315,"num_ops":10,"prepare.queue_time_us":254311,"prepare.run_cpu_time_us":18415,"prepare.run_wall_time_us":175706,"raft.queue_time_us":299571,"raft.run_cpu_time_us":36023,"raft.run_wall_time_us":239533,"replication_time_us":698181,"thread_start_us":557746,"threads_started":4,"wal-append.queue_time_us":356403,"wal-append.run_cpu_time_us":37437,"wal-append.run_wall_time_us":2165865}]]}
W20250626 08:24:28.102421 5470 log.cc:927] Time spent T 148557a00636474cad8673c1f02aae5e P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250626 08:24:28.102579 5420 leader_election.cc:290] T c8539685c775453b8aaf4921fa90fa63 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:28.103389 5286 leader_election.cc:290] T d606b2ba0bd74eea8cd6e0c6e109c5ac P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:28.053625 5271 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.107566 1920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39260 (request call id 2057) took 1239 ms. Trace:
I20250626 08:24:28.108098 1920 rpcz_store.cc:276] 0626 08:24:26.867910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:26.908211 (+ 40301us) service_pool.cc:225] Handling call
0626 08:24:26.925236 (+ 17025us) raft_consensus.cc:1469] Updating replica for 2 ops
0626 08:24:26.931950 (+ 6714us) raft_consensus.cc:1532] Early marking committed up to index 4
0626 08:24:26.963236 (+ 31286us) log.cc:844] Serialized 14 byte log entry
0626 08:24:28.034835 (+1071599us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0626 08:24:28.035576 (+ 741us) log.cc:844] Serialized 52 byte log entry
0626 08:24:28.035774 (+ 198us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:28.041137 (+ 5363us) log.cc:844] Serialized 14 byte log entry
0626 08:24:28.055377 (+ 14240us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:28.055452 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0626 08:24:28.107288 (+ 51836us) raft_consensus.cc:1683] finished
0626 08:24:28.107315 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:28.107494 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13906,"prepare.queue_time_us":51967,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":56746,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":15031,"thread_start_us":1023278,"threads_started":2,"wal-append.queue_time_us":576210}
I20250626 08:24:28.122347 1920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b498813b859f40cd95ed6adf6a2d8918" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:28.124106 2580 raft_consensus.cc:2466] T 84dfea36e4274e44b9a893967d69fd7e P be56082cacb54e9aa8f1a9ee91e70d76 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 10.
I20250626 08:24:28.129630 1923 raft_consensus.cc:2466] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 5.
I20250626 08:24:28.131836 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.134214 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75b98c18e48c4076ba1d545113f76522" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.135020 2000 raft_consensus.cc:2466] T 75b98c18e48c4076ba1d545113f76522 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:28.135459 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.136373 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.138468 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.132123 1890 leader_election.cc:304] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:28.081229 5380 raft_consensus.cc:3058] T 766debd72f3f4feaa9e81cd3690afdc1 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 08:24:28.045321 5434 raft_consensus.cc:513] T 9122458a1eae4f07b1a7f4ff63d6be19 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.078006 5368 raft_consensus.cc:491] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:28.154790 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "856b2d0e8d2948bd9bfc33345dcd4886" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.182770 1994 raft_consensus.cc:2366] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 for earlier term 5. Current term is 6.
W20250626 08:24:28.186009 1890 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 58218.9us
I20250626 08:24:28.187529 1997 raft_consensus.cc:2391] T 3946d60648894026984085c3abee51ea P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 8: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:28.107856 5435 raft_consensus.cc:2747] T a71e0323b96e4a47b1ba2e0a3d72840f P be56082cacb54e9aa8f1a9ee91e70d76 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250626 08:24:28.110225 5109 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.019s sys 0.020s
I20250626 08:24:28.193322 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0899df8ec53c45498dd1f3da415d5155" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.194017 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc905d51bc4a450484091ffa39379396" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.079893 5408 consensus_queue.cc:1046] T 736124bc239f4257b44dfc38e3acc17f P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20250626 08:24:28.091838 5472 raft_consensus.cc:2802] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 08:24:28.059317 5331 raft_consensus.cc:513] T 4a65a55472e64be4b9ae56426278c3c1 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:28.195844 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e262c3543404a9cb840169098735ea2" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.188050 5086 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.011s sys 0.022s
I20250626 08:24:28.116107 5317 raft_consensus.cc:491] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:28.072608 5418 consensus_queue.cc:1035] T 84dfea36e4274e44b9a893967d69fd7e P 548099d81e7f46ac9b2f03d9e5e678a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.070s
I20250626 08:24:28.201942 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9f49ece70144b92b201204afc3042ed" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.230664 1993 raft_consensus.cc:2466] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
W20250626 08:24:28.141768 5373 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.034s
W20250626 08:24:28.201340 2585 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.006s sys 0.039s
I20250626 08:24:28.121558 5420 raft_consensus.cc:2747] T 423807ecf2d54e869ffc86a272ebd10a P 755d83ca0913406b816e95415d502055 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250626 08:24:28.202224 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46151fddbbf14be68feb086aa1fee4d9" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.201850 5271 leader_election.cc:290] T 9996474aab3b4a5ca0e10b9f200b2abc P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 755d83ca0913406b816e95415d502055 (127.0.171.195:38293), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:28.073756 5343 raft_consensus.cc:513] T 9996474aab3b4a5ca0e10b9f200b2abc P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.208130 1923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.208091 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.242417 2002 raft_consensus.cc:2391] T 28139bf29a374c98a8fec4eddf814fac P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:28.221383 1994 raft_consensus.cc:1273] T f2bf9541d0634f9fab914afd74bb9505 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Refusing update from remote peer 74a4b66e1f7b4aed8fae26d31f9cc4e7: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250626 08:24:28.221961 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.200908 1921 raft_consensus.cc:2466] T 28139bf29a374c98a8fec4eddf814fac P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 5.
I20250626 08:24:28.221968 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac6c556e015045ac949f4ee3d2048b56" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.236466 1814 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 316052us
I20250626 08:24:28.243944 1998 raft_consensus.cc:2466] T ac6c556e015045ac949f4ee3d2048b56 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 6.
I20250626 08:24:28.244683 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cca900160913464799d0cfa9d2866779" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.245087 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.245664 1921 raft_consensus.cc:2391] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 in current term 4: Already voted for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in this term.
I20250626 08:24:28.246613 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edeafceae4424506a7bffa7a9fbac201" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:28.248531 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5549f2d4a32b4170ad59fc7489d1365f" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.249485 1994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7846e33f4b874a229eb7250f99a6db61" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.250885 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "171f89014c824d17a510a3cc17b3af0d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
I20250626 08:24:28.251487 1998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a71e0323b96e4a47b1ba2e0a3d72840f" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:28.258410 1994 raft_consensus.cc:1238] T 3d1eb5f502d8430b9109e246276cc44e P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250626 08:24:28.138554 5474 raft_consensus.cc:2762] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250626 08:24:28.228134 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.268710 2001 raft_consensus.cc:2466] T 4c17e5edca1f4c3784180a1653f3234d P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in term 7.
I20250626 08:24:28.229346 1995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.228906 1919 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.015s sys 0.029s
W20250626 08:24:28.205977 5109 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.024s sys 0.020s
W20250626 08:24:28.223343 5086 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.011s sys 0.022s
I20250626 08:24:28.106658 5445 raft_consensus.cc:513] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
W20250626 08:24:28.236070 2585 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.007s sys 0.039s
I20250626 08:24:28.154035 5214 consensus_queue.cc:1035] T bcf570bd4fbe46dc9453a7319f45a376 P 755d83ca0913406b816e95415d502055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.088s
I20250626 08:24:28.093433 5327 raft_consensus.cc:491] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:28.239724 1997 raft_consensus.cc:2466] T 46151fddbbf14be68feb086aa1fee4d9 P 755d83ca0913406b816e95415d502055 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a4b66e1f7b4aed8fae26d31f9cc4e7 in term 7.
W20250626 08:24:28.240039 1812 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 319253us
I20250626 08:24:28.283849 1812 leader_election.cc:304] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:28.240489 1927 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39254 (request call id 2183) took 1085 ms. Trace:
I20250626 08:24:28.251684 1921 raft_consensus.cc:2466] T 171f89014c824d17a510a3cc17b3af0d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d83ca0913406b816e95415d502055 in term 8.
I20250626 08:24:28.284734 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:28.285540 2000 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 548099d81e7f46ac9b2f03d9e5e678a6 in current term 5: Already voted for candidate 755d83ca0913406b816e95415d502055 in this term.
I20250626 08:24:28.285375 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.286302 1993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
W20250626 08:24:28.270123 1919 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.015s sys 0.029s
I20250626 08:24:28.270409 1994 pending_rounds.cc:85] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Aborting all ops after (but not including) 1
I20250626 08:24:28.195019 5368 raft_consensus.cc:513] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.238435 5472 raft_consensus.cc:695] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:28.278448 1816 leader_election.cc:304] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:28.279601 2001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf570bd4fbe46dc9453a7319f45a376" candidate_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.280215 1923 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:28.241510 5355 raft_consensus.cc:2747] T fec12ab89e754fea8f2b9e353bba3dee P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:28.282045 1814 leader_election.cc:304] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6, 74a4b66e1f7b4aed8fae26d31f9cc4e7; no voters:
I20250626 08:24:28.232036 5317 raft_consensus.cc:513] T cef3ff80065343cdbaf58ef8b1bbedf9 P be56082cacb54e9aa8f1a9ee91e70d76 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.304488 5269 raft_consensus.cc:513] T f2bf9541d0634f9fab914afd74bb9505 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:28.288687 1994 pending_rounds.cc:107] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250626 08:24:28.291347 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66ab2bec395d46f2a6a77385c304da93" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.317117 1849 raft_consensus.cc:2391] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 5: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
I20250626 08:24:28.318121 1994 raft_consensus.cc:2887] T d9f49ece70144b92b201204afc3042ed P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 08:24:28.298267 2002 raft_consensus.cc:1238] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [term 12 FOLLOWER]: Rejecting Update request from peer 548099d81e7f46ac9b2f03d9e5e678a6 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20250626 08:24:28.299327 5329 raft_consensus.cc:2747] T 148557a00636474cad8673c1f02aae5e P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250626 08:24:28.300027 5468 log.cc:927] Time spent T dc905d51bc4a450484091ffa39379396 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.194s user 0.000s sys 0.000s
I20250626 08:24:28.300122 5472 consensus_queue.cc:237] T 4c17e5edca1f4c3784180a1653f3234d P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:28.304899 1842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
W20250626 08:24:28.239531 5373 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.670s user 0.000s sys 0.034s
I20250626 08:24:28.305902 1921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2fc654780843abbda82ba8f5df515e" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.326511 1921 raft_consensus.cc:3058] T 5e2fc654780843abbda82ba8f5df515e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 7 FOLLOWER]: Advancing to term 8
I20250626 08:24:28.326855 1850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2717a332547d4ee8912bdda5311cffd8" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.306499 1816 leader_election.cc:304] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 548099d81e7f46ac9b2f03d9e5e678a6; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:28.294036 1927 rpcz_store.cc:276] 0626 08:24:27.155505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0626 08:24:27.857553 (+702048us) service_pool.cc:225] Handling call
0626 08:24:27.857890 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0626 08:24:27.858127 (+ 237us) raft_consensus.cc:1532] Early marking committed up to index 5
0626 08:24:27.871182 (+ 13055us) log.cc:844] Serialized 14 byte log entry
0626 08:24:28.240038 (+368856us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0626 08:24:28.240108 (+ 70us) raft_consensus.cc:1643] Marking committed up to 5
0626 08:24:28.240181 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0626 08:24:28.240270 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0626 08:24:28.240415 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15755,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"thread_start_us":368476,"threads_started":1}
I20250626 08:24:28.327670 1850 raft_consensus.cc:2391] T 2717a332547d4ee8912bdda5311cffd8 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 in current term 9: Already voted for candidate 548099d81e7f46ac9b2f03d9e5e678a6 in this term.
W20250626 08:24:28.225986 1965 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.008s sys 0.013s
I20250626 08:24:28.313047 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9996474aab3b4a5ca0e10b9f200b2abc" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.323763 1999 raft_consensus.cc:2366] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate be56082cacb54e9aa8f1a9ee91e70d76 for earlier term 4. Current term is 5.
W20250626 08:24:28.328354 1965 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.838s user 0.009s sys 0.013s
I20250626 08:24:28.328173 1844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c014cb521ac2424fa52882a7d676dbec" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.326659 1841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cef3ff80065343cdbaf58ef8b1bbedf9" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
W20250626 08:24:28.328686 1965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.196:37733, user_credentials={real_user=slave}} blocked reactor thread for 838043us
I20250626 08:24:28.329393 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.278097 5475 raft_consensus.cc:2802] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 FOLLOWER]: Leader election won for term 10
I20250626 08:24:28.328846 1844 raft_consensus.cc:2374] T c014cb521ac2424fa52882a7d676dbec P 548099d81e7f46ac9b2f03d9e5e678a6 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 755d83ca0913406b816e95415d502055 in term 9. Re-sending same reply.
I20250626 08:24:28.333087 1847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7a16389239e54c52bb619a62b1b0b91a"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:28.342216 1847 raft_consensus.cc:604] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Received request to transfer leadership to TS 74a4b66e1f7b4aed8fae26d31f9cc4e7
I20250626 08:24:28.342710 1847 raft_consensus.cc:612] T 7a16389239e54c52bb619a62b1b0b91a P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250626 08:24:28.328186 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" is_pre_election: true
W20250626 08:24:28.334203 5473 log.cc:927] Time spent T 02e74341f4bd4b68b89e1472c394f5bb P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20250626 08:24:28.345702 1967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.171.194:39813, user_credentials={real_user=slave}} blocked reactor thread for 70751.3us
I20250626 08:24:28.283080 5339 leader_election.cc:290] T 5697952652a3463cbe2ee01e18348697 P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:28.346460 1967 leader_election.cc:304] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters:
I20250626 08:24:28.346225 5411 leader_election.cc:290] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [CANDIDATE]: Term 8 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:28.354570 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28139bf29a374c98a8fec4eddf814fac" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:28.320392 5007 leader_election.cc:290] T 4be9d96212a6463eb1aa71d0b3d300a6 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 6 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
W20250626 08:24:28.324858 5436 log.cc:927] Time spent T 187f08b11130495ba8e2a482370447f4 P 74a4b66e1f7b4aed8fae26d31f9cc4e7: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250626 08:24:28.356312 5475 raft_consensus.cc:695] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [term 10 LEADER]: Becoming Leader. State: Replica: 74a4b66e1f7b4aed8fae26d31f9cc4e7, State: Running, Role: LEADER
I20250626 08:24:28.357486 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055"
I20250626 08:24:28.357345 5475 consensus_queue.cc:237] T 84dfea36e4274e44b9a893967d69fd7e P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.358757 2000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02e74341f4bd4b68b89e1472c394f5bb" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.360030 2002 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5697952652a3463cbe2ee01e18348697" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.360725 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fec12ab89e754fea8f2b9e353bba3dee" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.367166 2550 leader_election.cc:304] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56082cacb54e9aa8f1a9ee91e70d76; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055
I20250626 08:24:28.322535 5375 leader_election.cc:290] T 178e69a3d3834cae8db6684d32c1c4ce P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 12 election: Requested vote from peers 548099d81e7f46ac9b2f03d9e5e678a6 (127.0.171.193:43311), 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813)
I20250626 08:24:28.322518 5395 raft_consensus.cc:2802] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250626 08:24:28.371416 1997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8353d62d68914958b13bb87541c357f8" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "755d83ca0913406b816e95415d502055" is_pre_election: true
I20250626 08:24:28.378800 1997 raft_consensus.cc:2466] T 8353d62d68914958b13bb87541c357f8 P 755d83ca0913406b816e95415d502055 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56082cacb54e9aa8f1a9ee91e70d76 in term 8.
I20250626 08:24:28.368372 5412 raft_consensus.cc:491] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 08:24:28.373198 5445 leader_election.cc:290] T cca900160913464799d0cfa9d2866779 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 5 election: Requested vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), be56082cacb54e9aa8f1a9ee91e70d76 (127.0.171.196:37733)
I20250626 08:24:28.346537 5119 raft_consensus.cc:2802] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 FOLLOWER]: Leader election won for term 5
W20250626 08:24:28.389457 1893 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.171.195:38293, user_credentials={real_user=slave}} blocked reactor thread for 325062us
I20250626 08:24:28.378830 1849 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3946d60648894026984085c3abee51ea" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:28.390278 1893 leader_election.cc:304] T cca900160913464799d0cfa9d2866779 P 74a4b66e1f7b4aed8fae26d31f9cc4e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, 755d83ca0913406b816e95415d502055; no voters: be56082cacb54e9aa8f1a9ee91e70d76
W20250626 08:24:28.361661 5459 consensus_meta.cc:287] T 331d5bdb3fda41489445725e41b94bd1 P be56082cacb54e9aa8f1a9ee91e70d76: Time spent flushing consensus metadata: real 0.562s user 0.012s sys 0.000s
I20250626 08:24:28.383051 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
I20250626 08:24:28.391520 1847 raft_consensus.cc:2366] T 4c17e5edca1f4c3784180a1653f3234d P 548099d81e7f46ac9b2f03d9e5e678a6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 755d83ca0913406b816e95415d502055 for earlier term 7. Current term is 8.
I20250626 08:24:28.312501 5327 raft_consensus.cc:513] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } } peers { permanent_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" member_type: VOTER last_known_addr { host: "127.0.171.196" port: 37733 } attrs { promote: false } }
I20250626 08:24:28.380237 2551 leader_election.cc:304] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d83ca0913406b816e95415d502055, be56082cacb54e9aa8f1a9ee91e70d76; no voters:
I20250626 08:24:28.386750 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c17e5edca1f4c3784180a1653f3234d" candidate_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.375811 5271 raft_consensus.cc:2747] T 2e6a8a739b954cc082a290e6c529a271 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250626 08:24:28.396729 1965 leader_election.cc:304] T 0899df8ec53c45498dd1f3da415d5155 P 755d83ca0913406b816e95415d502055 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 755d83ca0913406b816e95415d502055; no voters: 74a4b66e1f7b4aed8fae26d31f9cc4e7, be56082cacb54e9aa8f1a9ee91e70d76
I20250626 08:24:28.388823 5482 raft_consensus.cc:2802] T 856b2d0e8d2948bd9bfc33345dcd4886 P 755d83ca0913406b816e95415d502055 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 08:24:28.369937 5007 raft_consensus.cc:2747] T 02e74341f4bd4b68b89e1472c394f5bb P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250626 08:24:28.408106 1847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f8cb31b10924fbcb39dfbbcac77397e" candidate_uuid: "755d83ca0913406b816e95415d502055" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" is_pre_election: true
I20250626 08:24:28.410281 1847 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dc905d51bc4a450484091ffa39379396"
dest_uuid: "548099d81e7f46ac9b2f03d9e5e678a6"
mode: GRACEFUL
new_leader_uuid: "755d83ca0913406b816e95415d502055"
from {username='slave'} at 127.0.0.1:50658
I20250626 08:24:28.414487 1927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4be9d96212a6463eb1aa71d0b3d300a6" candidate_uuid: "be56082cacb54e9aa8f1a9ee91e70d76" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7"
I20250626 08:24:28.377796 5395 raft_consensus.cc:491] T 3e928895bcfa4d00823f3f4b50eebf86 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 755d83ca0913406b816e95415d502055)
I20250626 08:24:28.361400 5408 consensus_queue.cc:1046] T a71e0323b96e4a47b1ba2e0a3d72840f P 755d83ca0913406b816e95415d502055 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20250626 08:24:28.389489 5412 raft_consensus.cc:513] T 0914c29163894ada9540f1133a461648 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "548099d81e7f46ac9b2f03d9e5e678a6" member_type: VOTER last_known_addr { host: "127.0.171.193" port: 43311 } } peers { permanent_uuid: "74a4b66e1f7b4aed8fae26d31f9cc4e7" member_type: VOTER last_known_addr { host: "127.0.171.194" port: 39813 } } peers { permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195" port: 38293 } }
I20250626 08:24:28.419193 5109 raft_consensus.cc:2802] T 8353d62d68914958b13bb87541c357f8 P be56082cacb54e9aa8f1a9ee91e70d76 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250626 08:24:28.411641 5119 raft_consensus.cc:695] T 66ab2bec395d46f2a6a77385c304da93 P 548099d81e7f46ac9b2f03d9e5e678a6 [term 5 LEADER]: Becoming Leader. State: Replica: 548099d81e7f46ac9b2f03d9e5e678a6, State: Running, Role: LEADER
I20250626 08:24:28.399377 5327 leader_election.cc:290] T 4f29b6256843489397f6e58f34751764 P be56082cacb54e9aa8f1a9ee91e70d76 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 74a4b66e1f7b4aed8fae26d31f9cc4e7 (127.0.171.194:39813), 755d83ca0913406b816e95415d502055 (127.0.171.195:38293)
I20250626 08:24:28.366230 5435 consensus_queue.cc:1035] T 75b98c18e48c4076ba1d545113f76522 P be56082cacb54e9aa8f1a9ee91e70d76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d83ca0913406b816e95415d502055" member_type: VOTER last_known_addr { host: "127.0.171.195"